👨‍💻 Professional notebooks 👨‍🚀

Hunter’s haphazard hoard of hopefully helpful hacks

Algorithms | Athena workstations at MIT | AWS | C | Catholicism | Cloud | Databases | Distributed systems | Docker | Finances | Future topics | GDB | Games | Golang | Grammar and style | Housing | IntelliJ | Interviews | Julia | Kubernetes | LaTeX | Linear algebra | Linux | Literature | Mac | Machine learning | Miscellanea | Networks | ONC RPC | Operating systems | Pandoc | Probability | Projects | Python | Raspberry Pi | Screen | Security | Serialization | Shell | Software engineering | SSH | Strength training | System design | Tmux | Version control | Vim | Virtual machines | Visual Studio Code | Visualize C call graphs on a Mac | Web programming | Wolfram Alpha

Finances

Salary information

• Information and comparison (levels.fyi)

Equity compensation

• Overview
• Tax implications

  

AWS

General

• AWS services explained in one line each
• Overview of AWS services (10 hour video)
• AWS regions and corresponding AZs
  • aws ec2 describe-availability-zones --region us-west-1 check currently-available AZs for region
• Tools
  • cloud-nuke delete all resources in an AWS account, optionally by region

VPC

• Overview

  

  

EC2

• Instance type naming conventions

  

• Instance types

  

• Purchasing options

  

IAM

• Identities (users, groups, roles)

  

Kubernetes

General

• Reference
  • Homepage
  • Cheat sheet
  • Concepts
  • Debugging applications
  • Recommended labels
• Tools
  • kubectx + kubens easily switch contexts and namespaces
  • Popeye quickly profile cluster
  • Stern tail logs from multiple pods
  • helm-docs auto-generated Helm chart readmes
• Snippets
  • kubectl get secret orc8r-certs -o go-template='{{range $k,$v := .data}}{{printf "%s:\n" $k}}{{if not $v}}{{$v}}{{else}}{{$v | base64decode}}{{end}}{{"\n"}}{{end}}' print a map of opaque secrets
• Miscellanea
  • Controller vs. operator: controller is generic reconciliation loop (e.g. blue-green deployment), operator is a controller that manages a particular application (e.g. Postgres operator)

API server

• Object identifiers
  • Type: specific Go struct defining the schema of an object, e.g. Pod struct
  • Version: point-in-time identifier for representation of a type, e.g. v1 or v1beta1
  • Kind: unique name for the type within the version, e.g. Pod or Service
  • Group: set of related versions, kinds, and types that evolve together over time, e.g. core or apps
  • API Version: combination of group and version to specify the API to target, e.g. v1 (i.e. core/v1) or apps/v1beta1

    

• Object versioning
  • Overview: object updates always update resource version, and only update generation if the update changes the spec
  • Resource Version: opaque versioning string for optimistic concurrency control (backed by etcd revisions)
  • Generation: sequence number incremented when the object’s desired state is changed

Scheduling

• Pod scheduling cycle

  

• Resource requests and limits

  

Node runtime

• Container runtime landscape

  

• Containerd architecure

  

Scale

• Scale limits for single cluster
  • 100 pods per node
  • 5k nodes
  • 150k pods
  • 300k containers

DNS resolution

• DNS-based service discovery spec
• Service-based DNS lookup path

  

EKS

• Architecture

  

Docker

Containers

• About (wikipedia)
• What is a container: namespaces and cgroups
  • And union mounts (intro) (history)
• Overview (playlist)
• Container vs. VM

  

• Container vs. unikernel (about)

  

Reference

• Dockerfile reference
  • Prefer COPY over ADD
  • Build cache rules overview
    • ADD and COPY only commands to dynamically invalidate
    • ARG change invalidates line of first use
    • RUN only invalidates if command to run changes
• Logging overview – output of container’s PID 1 is forwarded to host as the application’s logs
• Docker-compose networking
• Tools
  • lazydocker: TUI for Docker
  • dive: explore Docker image and layer contents
  • gosu: Docker-targeted su/sudo implementation for dropping a process’s privilege level

Snippets

• docker network inspect -f '{{json .Containers}}' orc8r_default | jq '.[] | .Name + ":" + .IPv4Address' print all IP addresses in a Docker network
• docker cp ~/Downloads/out.sql orc8r_postgres_1:/ copy file to Docker container
• docker exec -it orc8r_maria_1 mysql -u magma_dev -p exec command with args in container (e.g. bash)
• docker ps list containers
• docker system df what’s available to prune
• docker system prune docker doesn’t perform garbage collection
• docker-compose logs -f controller* follow logs for controller container
• docker-compose ps list containers (in context of docker-compose.yml)
• docker-compose ps --services list services

Runtime privilege and Linux capabilities

Default granted - AUDIT_WRITE write records to kernel auditing log - CHOWN make arbitrary changes to file UIDs and GIDs (see chown(2)) - DAC_OVERRIDE bypass file read, write, and execute permission checks - FOWNER bypass permission checks on operations that normally require the file system UID of the process to match the UID of the file - FSETID don’t clear set-user-ID and set-group-ID permission bits when a file is modified - KILL bypass permission checks for sending signals - MKNOD create special files using mknod(2) - NET_BIND_SERVICE bind a socket to internet domain privileged ports (port numbers less than 1024) - NET_RAW use RAW and PACKET sockets - SETFCAP set file capabilities - SETGID make arbitrary manipulations of process GIDs and supplementary GID list - SETPCAP modify process capabilities - SETUID make arbitrary manipulations of process UIDs - SYS_CHROOT use chroot(2), change root directory

Default not granted - AUDIT_CONTROL enable and disable kernel auditing; change auditing filter rules; retrieve auditing status and filtering rules - AUDIT_READ allow reading the audit log via multicast netlink socket - BLOCK_SUSPEND allow preventing system suspends - BPF allow creating BPF maps, loading BPF Type Format (BTF) data, retrieve JITed code of BPF programs, and more - CHECKPOINT_RESTORE allow checkpoint/restore related operations. Introduced in kernel 5.9 - DAC_READ_SEARCH bypass file read permission checks and directory read and execute permission checks - IPC_LOCK lock memory (mlock(2), mlockall(2), mmap(2), shmctl(2)) - IPC_OWNER bypass permission checks for operations on System V IPC objects - LEASE establish leases on arbitrary files (see fcntl(2)) - LINUX_IMMUTABLE set the FS_APPEND_FL and FS_IMMUTABLE_FL i-node flags - MAC_ADMIN allow MAC configuration or state changes. Implemented for the Smack LSM - MAC_OVERRIDE override Mandatory Access Control (MAC). Implemented for the Smack Linux Security Module (LSM) - NET_ADMIN perform various network-related operations - NET_BROADCAST make socket broadcasts, and listen to multicasts - PERFMON allow system performance and observability privileged operations using perf_events, i915_perf and other kernel subsystems - SYS_ADMIN perform a range of system administration operations - SYS_BOOT use reboot(2) and kexec_load(2), reboot and load a new kernel for later execution - SYS_MODULE load and unload kernel modules - SYS_NICE raise process nice value (nice(2), setpriority(2)) and change the nice value for arbitrary processes - SYS_PACCT use acct(2), switch process accounting on or off - SYS_PTRACE trace arbitrary processes using ptrace(2) - SYS_RAWIO perform I/O port operations (iopl(2) and ioperm(2)) - SYS_RESOURCE override resource Limits - SYS_TIME set system clock (settimeofday(2), stime(2), adjtimex(2)); set real-time (hardware) clock - SYS_TTY_CONFIG use vhangup(2); employ various privileged ioctl(2) operations on virtual terminals - SYSLOG perform privileged syslog(2) operations - WAKE_ALARM trigger something that will wake up the system

Tcpdump

• Overview
• docker exec orc8r_controller_1 tcpdump -U -w - 2>/dev/null -- > controller.pcap tcpdump to file on host machine
• Capture container network traffic, from host, with no extra per-container installs
  • Create container to capture traffic sh docker build -t tcpdump - <<EOF FROM ubuntu RUN apt-get update && apt-get install -y tcpdump CMD tcpdump -i eth0 EOF
  • docker run -i --net=container:orc8r_controller_1 tcpdump tcpdump -U -w - 2>/dev/null -- > controller_and_proxy.pcap capture directly onto host

Cloud

Prometheus

• Snippets
  • {__name__=~".+"} query all non-stale metrics
• Architecture

  

Terraform

• Reference
  • Intro
  • Official tutorials
  • Debugging: set TF_LOG environment variable to one of TRACE, DEBUG, INFO, WARN, ERROR
• Tools
  • tfenv Terraform version manager
  • terraform-docs generate module documentation
• Snippets
  • chomp(file("~/.magma/github_access_token")) remove newline from file input

Elasticsearch

• Architecture

  

• Scale limits for single domain (AWS)
  • 3 PB storage (default limit is 600 TB)
  • 200 instances (default limit is 40 instances)

Swagger

• Tools
  • Swag generate swagger.yml from godoc comments
  • go-swagger better generated Go code
    • Model generation rules
    • swagger mixin merge Swagger specs using go-openapi/analysis

Interviews

Overview

Interview types

Interviews generally fall into 1 of 3 categories

• Coding: design and code up a solution to a single problem (coding pad)
  • DS&A: most coding interviews will require data structures and algorithms (DS&A) insights to complete
  • Practical: a minority of coding interviews will just focus on practical, day-to-day coding skills
  • Frontend: if you’re going for a frontend role, you will likely be challenged to build a UI component of some sort
• Design: explore and design a high-level architecture for an under-specified problem (virtual whiteboard)
  • System: most design interviews will require designing a scalable, efficient, robust distributed system
  • Object-oriented: some design interviews will focus on the design of a single, in-process component
  • Product: for more frontend or product management type roles, the focus will be on designing an API, and what UX to provide
• Behavioral: describe approaches and past experiences (verbal discussion)

Additional notes

• Timing: most companies will do 1 hour per interview, where 50-55 minutes are devoted to the actual interview; some however will only allot 45 minutes per interview, where 35-40 minutes are devoted to the interview (e.g. Meta)
• Number of questions/parts: most companies only expect 1 question/part per interview; some however expect multiple (potentially completely separate) questions within a single interview (e.g. Meta expects 2 separate questions during coding interviews)
• Split: behavioral questions will mostly be standalone in their own interview; increasingly I’ve started seeing the behavioral component split out into the first 10-15 minutes of each of the other interviews
• Idiosyncratic interview types
  • Bug squash: set up an IDE in a repo and resolve as many bugs as you can (e.g. Stripe)
  • Technical deep dive: this can either be a discussion-only system design interview (e.g. LinkedIn), or a project retrospective where you present the technical details of a past project (e.g. OpenAI)
  • Project design: read a project’s technical requirements, then write up a design doc-style one pager on your design recommendations (e.g. Waymo)
  • Technical trivia: answer trivia questions on a technical topic (e.g. Jump Trading)

Tools

• Use levels.fyi to set level and compensation expectations per company
• Design interview tools (to move faster / save time)
  • I recommend defaulting to Excalidraw and using the largest iPad Pro you can buy, with an Apple Pencil, as an additional display (it can be a little buggy, but I personally still like it)
  • At minimum, use a mouse, rather than a trackpad

Remote-friendly

Remote-friendly big-tech companies (as of January 2024)

• Meta (IC6 and above)
• Microsoft
• Netflix
• Pinterest
• Stripe

Prep

Rubrics

• Coding
  • Communication
  • Problem solving
  • Coding
  • Verification
• Design
  • Exploration
  • Handling data
  • Component responsibilities
  • Completeness
  • Tradeoffs
  • Quantitative
  • Deep dive
• Behavioral
  • Leadership
  • Problem solving
  • Teamwork
  • Communication
  • Impact
  • Values

Coding

• LeetCode top 150

Design

• Overview
• Concept walkthrough
• Deep-dive design examples
• See the system design section below for more info/examples
• Response template
  • Explore (5 min)
  • Estimate (5 min)
  • Think (2 min)
  • Design goals (2 min)
  • High-level design (5 min)
  • Deep dive (20 min)
  • Wrap-up (walkthrough, and if time: further considerations, additional trade-offs (5 min)

Behavioral

• STAR method: Situation, Task, Action, Results
  • Organize responses using this paradigm
• Ask ChatGPT to be a behavioral interviewer
  • For the target company and level
  • Tell it to give feedback on each response: score out of 100 and how could improve
  • Use speech-to-text mode – you talk, it transcribes then responds with text

System design

Specific problems

• Rate limiting (intro) (distributed)
• Load balancing (intro)
• Full text search (SSTable + LSM tree)

Partitioning and replication

• Partitioning techniques
  • Consistent hashing: place bins and balls onto a circle (with virtual bins)

    

  • Rendezvous hashing: compute w(i, j) = \text{hash}(\text{ball}(i), \text{bin}(j)), assign ball to the k bins with highest weights

    

  • Optimal placement (e.g. Meta’s Shard Manager): place shards according to application-reported metrics and goals, look up shard location in consistent datastore

    

Latency

• Latency estimates

  

• Frontend latency estimates

Software engineering

Overview

• Career framework (and comparable framework from Dropbox)

  

Decision making

• Steps toward making a decision
  1. Assess the situation (understand initial question + facts)
  2. Get input (clarify key question + assemble factbase)
  3. Develop alternatives
  4. Recommend an alternative
  5. Decide
  6. Communicate decision
  7. Execute
• RAPID framework: each letter is a specific role, with the Decider as the DRI

  

  

Development

• High quality software is worth the cost (Martin Fowler)
• How to write a design doc
• Productive remote development
  • iTerm2 rather than Terminal.app
  • Eternal Terminal rather than SSH
  • tmux in native mode, e.g. alias e0="et -c 'tmux -CC new-session -AD -s main' -x devvm7602.ftw0.facebook.com:8080"
  • Remote pbcopy to copy from remote to local
  • iTerm2 triggers to send local desktop notifications on remote command completions, e.g. with this noti command

Collaboration

• Wandbox: online IDE for many languages and versions, with shareable permalinks
• CodeWithMe: IntelliJ IDE sharing
• tmate: terminal session sharing

User experience

• Errors and user control
  • Mistake (error in planning or rule application), lapse (failure of memory), slip (failure of execution)
  • Capture slip: started the same way, veered off into wrong action set
  • Description slip: chose wrong action set, because it’s similar to the desired action set

    

Patterns

• General
  • Modern software development (12-factor app)
  • Level-triggered vs. edge-triggered (state vs. event)
  • Broker vs. brokerless architectures
  • Dependency injection (DI) allows a caller to supply the dependencies of a callee, supporting e.g. easier testing
  • Data access object (DAO) abstract, often restricted interface around a database or other persistence mechanism
• Domain-driven design
  • Domain primitives
    • Set domain primitives, e.g. Quantity or DeliveryDays as an object with validation
    • Perform validation either on creation or as early as possible, e.g. at API boundaries
    • Result in “if an object exists, it contains a valid value”
• Security
  • Payload validation order: origin, size, lexical content, syntax, then lastly semantics
• API design
  • List pagination: page size, page token, and next page token
• Testing
  • Nil package-level function run during a method, to be set by testing functions, allowing testing functions to run custom hooks during the real method
• Code quality
  • 80-character line lengths aren’t optimal anymore

Culture

• The Jargon File
• New Programming Jargon

Concurrency

• Intro
• Problems with asynchronous programming (What Color is Your Function?)

Typing

• Static vs. dynamic typing: static checks types before run-time, dynamic check types on-the-fly during run-time
• What’s wrong with null: non-composable, requires dynamic vs. static type checking

CI/CD

• Continuous delivery from a single branch (FB)

Package management

• Lock files allow deterministic installation of dependencies (about) (stackoverflow)
  • E.g. package-lock.json
  • Lock files should be committed to version control

Computation structures

• Floating point numbers
  • Float playground

    

Tools

• Code quality
  • LGTM static analysis on PRs
  • Codacy configurable linting on PRs
  • Codecov test coverage
  • golangci-lint Go linters aggregator
• Code navigation
  • Sourcegraph universal code search, with code coverage plugins
  • Sourcetrail source code visualization
• Security
  • SOPS store encrypted secrets in version control
• Packaging
  • goreleaser package and deliver Go binaries
• Utilities
  • csvtomd convert CSV file to Markdown table
  • Regex101 full-featured regex tester

Libraries

• Go
  • Trending Go repositories
  • List of Go projects (Awesome Go)
  • Afero file system abstraction
  • Cobra CLI generation
  • Ent ORM for graph-based entities
  • Errors extends standard errors library with support for context wrapping
  • Gorm default ORM
  • Sqlx extensions to database/sql
  • Viper configuration
• Python
  • List of Python projects (Awesome Python)
  • CFFI interface with C/C++ code
  • Click CLI generation
  • Fabric execute shell commands remotely over SSH (intro)
  • MonkeyType generate static type annotations by collecting runtime types
  • Nubia interactive shell generation
  • NumPy storage and operation on large, multi-dimensional arrays
    • About (home) (reference)
    • Visual guide
    • Tutorial
    • Cheat sheets (pdf) (html)
    • Operators
      • A @ B matrix multiplication of A by B
  • Swig generate bindings for C/C++ code
  • XAR package many files into a single self-contained executable file

SSH

• Intro (wikipedia) (overview) (examples)
• Resources
  • Illustrated guide
  • OpenSSH wiki
  • Set up enterprise-level SSH environment, with certificates and proxies
• ssh command
  • -vvv max verbosity
  • ssh vagrant@localhost -p 2222 specify non-standard port
  • Vagrant-specific
    • vagrant ssh-config ssh configuration for each VM, e.g. which private key to use
    • vagrant ssh cwag_test -- -vvv send SSH options through vagrant command
• Configuration
  • System-wide config files
    • ~/.ssh/known_hosts public key info for hosts known to the client (file format) (key format)
      • Fields: markers (optional), hostnames, keytype, base64-encoded key, comment, space-delimited
      • Keytype may be further decomposed, e.g. into bits, exponent, modulus
  • Connection config sources, by order of precedence (common options)
    • Command line-specified
    • ~/.ssh/config
    • /etc/ssh/ssh_config
  • Config for transient servers, e.g. local VMs
    • UserKnownHostsFile=/dev/null don’t pollute user’s known_hosts file
    • StrictHostKeyChecking=no don’t enforce known_hosts entries
    • IdentityFile indicate which private key to use for host
• Public key access (howto)
  • Create public-private keypair
  • Install non-secret public key in ~/.ssh/authorize_keys on target server
  • Indicate location of private key on local host, if it’s in a non-standard location
• Message ID values

  

IntelliJ

• IntelliJ IDEA
• IdeaVim, Vim emulation for IntelliJ IDEs
• Set up automatic code formatting (Go)
• Fixes
  • Prevent auto-closing tabs: Settings > Editor > General > Editor Tabs > Tab limit
  • Remote interpreter will choose cached version of files, workaround is to add local version of the files
• Easy actions
  • Contextual menu > implement interface
• Keymaps
  • Miscellanea
    • shift shift omni-action
    • <alt-enter> contextual menu
    • <ctrl-tab> switcher
    • ctrl ctrl run anything
    • <cmd-up> navigation bar
  • Remote targets
    • <cmd-alt-y> sync file
    • Invalidate Caches / Restart nuke caches
  • Go to
    • <cmd-b>, <ctrl-j> declaration
    • <cmd-shift-b> type declaration
    • <cmd-k> implementations
    • <cmd-shift-k> implements
    • <cmd-j> show usages
    • <cmd-shift-j> find usages
    • f2, <ctrl-;> next error
    • <shift-f2>, <ctrl-'> previous error
    • <ctrl-shift-;> next change
    • <ctrl-shift-'> previous change
    • <ctrl-.> next method
    • <ctrl-,> previous method
  • View
    • <shift-esc> hide active tool window
    • <cmd-shift-'> maximize tool window
    • <cmd-shift-e> recent locations window
    • <cmd-3> find window
    • <cmd-e> open recent files
    • <cmd-t> terminal window
    • <cmd-o> reveal in file tree
    • <cmd-+> expand all in project menu
    • <cmd-shift--> close all folds
    • <cmd-shift-=> expand all folds
    • <cmd-y> quick definition
    • <cmd-shift-a> version control annotations
    • <cmd-shift-d> file/package structure
  • Search
    • !*.go exclude-based file mask
    • right click > mark directory as > excluded exclude specific directories
  • Edit
    • <shift-f6> refactor
    • <shift-f6> <shift-f6> refactor dialogue (e.g. disable string and comment refactor)
    • <shift-n> create new file
    • <c-n> next autocomplete option
    • <c-p> previous autocomplete option
    • <cmd-l> roll back changed hunk
  • Run
    • <cmd-r> run
    • <cmd-shift-r> rerun
  • Move
    • <cmd-shift-t> open tests
    • <alt-tab> cycle split
    • <alt-shift-tab> cycle previous split
    • <ctrl-cmd-n> split vertically
    • <ctrl-cmd-m> split horizontally
    • <ctrl-cmd-z> unsplit
    • <ctrl-cmd-y> unsplit all
    • <ctrl-cmd-o> move tab to opposite split

Serialization

Intro

• Protobufs vs. flatbufs vs. other
• Some benchmarks

Some formats

• Protocol buffers (protobufs) generically usable
• FlatBuffers (flatbufs) zero-parse
• JSON (JavaScript Object Notation) untyped, non-binary
• ASN.1 (Abstract Syntax Notation One) multiple encodings, canonical encoding available

Protocol buffers

• Message definition forward compatibility
• Encoding is ID-tag-value, requiring protobuf descriptor to correctly deserialize
  • ID unambiguously defines the type of the field by referencing the associated protobuf definition
  • Tag values (wire types) determine length of the field
    • 0 varint – int32, int64, uint32, uint64, sint32, sint64, bool, enum
    • 1 64-bit – fixed64, sfixed64, double
    • 2 length-delimited – string, bytes, embedded messages, packed repeated fields
    • 5 32-bit – fixed32, sfixed32, float
  • Value is encoded depending on the type
    • Varint fields are encoded least-significant bits first, with the MSB of each non-terminal byte as 1, else 0
    • Length-delimited fields are encoded as length-value
    • 64-bit and 32-bit fields are fixed-width
• Style guides (Google) (Uber)
  • Common issues
    • Consistent service namings
  • Example file
  • File structure
    • License
    • File overview
    • Syntax
    • Package
    • Imports (sorted)
    • File options
    • Everything else
• Go
  • protobuf/ptypes/wrappers provides wrapper structs for Go built-in types, e.g. StringValue, BytesValue, etc.
  • proto.Equal(protoA, protoB) compare equality of protobufs
  • proto.Clone(data).(*protos.SubscriberData) clone a protobuf
  • Sending any.Any
  • Alternative gRPC encodings and compressors

gRPC

• Awesome gRPC gRPC and protobuf resources
• Streaming RPCs strongly discouraged (about) (40m video)
• protoc --go_out=plugins=grpc:. *.proto generate gRPC-supported protobuf code
• Go
  • Hello world
    • go mod init github.com/hcgatewood/any-json init Go modules to import third-party and cross-package code in toy project
  • Returning nil, errors.New("zebra") for a response results in the specified error returned to client
  • Returning nil, nil for a response results in an error (error while marshaling) returned to client, even if the return message is defined as empty

Canonical forms

• Overview
  • Seems like the best solution to deriving a usably-canonical form (aside from defaulting to ASN.1) is picking a particular JSON canonicalizing format and accordingly converting relevant objects to JSON
  • Protobufs support a canonical encoding into JSON, so this seems like the best way to convert protobufs into a canonical representation
• Protobufs
  • Protobufs provide no cross-language canonical serialization form
  • Some languages (e.g. Go, C++) provide deterministic, but not canonical, serialization
  • One of a set of issues blocking a canonical format is map key-value pair ordering is undefined
    • In Go, map ordering seems randomized across serialization invocations
• JSON
  • No single, RFC-like standard describing a canonical JSON form
    • matrix.org (Python)
    • gibson042 (Go)
  • Seems like Go’s Protobuf JSON encoder jsonpb produces more-or-less canonical JSON
  • Seems like Python’s general JSON encoder json can be coerced into producing more-or-less canonical JSON

Literature

• Great books
  • Meta list of great books lists
  • Mortimer Adler’s recommended reading list, from How to Read a Book
  • Harold Bloom’s western canon, splits into four ages: theocratic, aristocratic, democratic, and chaotic
• Other reading lists
  • List of joint winners of Hugo and Nebula awards (science fiction)
  • List of reading suggestions (Ask HN)
  • What books changed the way you think about almost everything (Ask HN)
• Miscellanea
  • NASA’s science fiction space technology terms

Catholicism

• List of popes
• Religious orders
  • List of religious orders (by papal production)

Games

Traditional sports

• Football overview

Factorio

• Factorio cheat sheet

Scrabble

• Official dictionary
• Two-letter words (full list) (with definitions)
  • AA, AB, AD, AE, AG, AH, AI, AL, AM, AN, AR, AS, AT, AW, AX, AY
  • BA, BE, BI, BO, BY
  • DA, DE, DO
  • ED, EF, EH, EL, EM, EN, ER, ES, ET, EW, EX
  • FA, FE
  • GI, GO
  • HA, HE, HI, HM, HO
  • ID, IF, IN, IS, IT
  • JO
  • KA, KI
  • LA, LI, LO
  • MA, ME, MI, MM, MO, MU, MY
  • NA, NE, NO, NU
  • OD, OE, OF, OH, OI, OK, OM, ON, OP, OR, OS, OW, OX, OY
  • PA, PE, PI, PO
  • QI
  • RE
  • SH, SI, SO
  • TA, TE, TI, TO
  • UH, UM, UN, UP, US, UT
  • WE, WO
  • XI, XU
  • YA, YE, YO
  • ZA

Housing

General

• Useful
  • List of places to live (Ask HN)
  • City guides (Wikivoyage)
  • Rent calculator
  • Square feet visualizer
  • When moving to a new location, look into getting a third-party company to provide rental tours
• Apps to make life easier
  • Instacart grocery delivery
  • Getaround: car share
  • Transit: live public transportation trip planning, including bikes and electric scooters
  • Lemonade: renter’s insurance

San Francisco

• Recreation
  • Where to play basketball
• Relevant maps

  

  

  

  

  

Linear algebra

Resources

• Introduction to Applied Linear Algebra by Boyd, Vandenberghe (“VMLS”)
  • Corresponding Julia Language Companion
• Introduction to Linear Algebra, fifth edition by Strang (“Strang”)
  • Corresponding lecture series
• Mathematics for Computer Science by Lehman, Leighton, Meyer (“6.042”)
  • Corresponding lecture series
• Matrices cookbook
• Applications of linear algebra
  • Original Google Page Rank algorithm
• Further study
  • Linear algebra
    • Further linear algebra study
  • Abstract algebra
    • Lecture series (Harvard Mathematics 122)

Topics

• Associativity of matrix multiplication
• Dot product intuition
• Orthogonality implies linear independence, but not the other way around
• Disambiguation of “affine” vs. “linear”
• Projections (intro) (as linear transformation)
• Eigenvalue visualization
• Volume of a generalized cone
• Common sets and number types
  • \emptyset = \{\} the empty set
  • \mathbb{N} = \{0, 1, 2, \ldots\} nonnegative integers aka natural numbers (sometimes not including zero)
  • \mathbb{Z} = \{\ldots, -2, -1, 0, 1, 2, \ldots\} integers
  • \mathbb{Q} \supset \{\frac{1}{2}, -\frac{5}{3}, 16\} rational numbers
  • \mathbb{R} \supset \{\pi, e, -9, \sqrt{2}\} real numbers
  • \mathbb{C} \supset \{i, \frac{19}{2}, \sqrt{2}-2i\} complex numbers
  • Modifiers
    • The + and - superscripts restrict a set to its positive and negative elements, respectively
    • \mathbb{Z^*} = \{0, 1, 2, \ldots\} nonnegative integers
• Inverse of a 2x2 matrix

  

• Eigenvalue equalities \begin{aligned} \prod \lambda_i &= \det A \\ \sum \lambda_i &= \sum a_{ii} \end{aligned}
• Projections \begin{aligned} A\hat{x} &= b \\ \hat{x} &= (A^TA)^{-1}A^{T}b \\ p = A\hat{x} &= A(A^TA)^{-1}A^{T}b \\ P &= A(A^TA)^{-1}A^{T} \\ P^2 &= P \\ P^T &= P \\ A \text{ invertible} &\implies P = I \\ |P| &\in \{0, 1\} \end{aligned}
• Inverse of matrix product \begin{aligned} (B^{-1}A^{-1})(AB) &= B^{-1}(A^{-1}A)B \\ &= B^{-1}B \\ &= I \\ \tilde{\implies} (AB)^{-1} &= B^{-1}A^{-1} \end{aligned}
• Cumulative moving average \begin{aligned} a_{n+1} &= \frac{x_{n+1} + nc_n}{n+1} \\ \iff a_{n+1} &= a_n + \frac{x_{n+1} - a_n}{n+1} \end{aligned}
• The four fundamental subspaces

  

• Set relations and binary relations
  • Function: right-unique and left-total
  • Injective aka one-to-one: left-unique, may be a function
  • Surjective aka onto: right-total, may be a function
  • Bijective: injective and surjective, may be a function

• Convex vs. affine combinations
  • Affine combination: sum of coefficients = 1; coefficients may be negative
  • Convex combination (“weighted average”): same, but coefficients >= 0

    

Julia

Reference

• Cheat sheat
• Linear algebra module
• Julia vs. Python vs. MATLAB

Snippets

• using LinearAlgebra; Diagonal(v) diagonal matrix from vector v

Jupyter notebooks

• “Jupyter” is an amalgam of “Julia”, “Python”, and “R”
• About (home) (overview) (docs) (wikipedia)
• Math equation rendering uses MathJax (usage)
• Getting started
  • Local

    • Install Python: brew install python3

    • Install Jupyter: pip3 install jupyter

    • Install Julia: brew cask install julia

    • Install IJulia, the Julia kernel for Jupyter notebooks (other kernels)

      $ julia
      julia> using Pkg
      julia> Pkg.add("IJulia")

    • Start Jupyter server: jupyter notebook (from relevant directory)

    • Print as PDF: jupyter nbconvert --to pdf in.ipynb

  • Hosted
    • Use JuliaBox (docs)

Strength training

General

• Intro and fundamentals
• Beginning resources and general guidelines
• Common abbreviations
• Reading food labels

Form

• All
  • Appropriately brace for each set/rep
• Deadlift
  • Intro (Jeff Nippard)
  • Avoid starting with hips too low
  • Avoid rounding back during lift (push floor away)
  • Avoid swaying the bar during lift (engage lats)
  • Avoid jerking the bar (lift upper back)
  • Avoid overextending during lockout
• Hip thrust (barbell)
• Overhead press
  • Flex glutes
• Squat
  • Allow knees to go past toes
  • Explode out of the hole

Fixes

• Posture support
  • Rounded shoulders
    • Stretch minor pec
    • Wall shoulder girdle stretch
    • Strengthen pull motion (rowing exercises)
  • Lordosis (anterior pelvic tilt)
    • Strengthen core, glutes, hamstrings
    • Stretch hip flexors (e.g. using bench)
  • Forward head posture
    • Isometric lying neck retraction: lie on back with feet flat on ground, then hold a push of the head into mat; relax and repeat

Macro tracking

• Lifesum

Kinesiology

• Kinesiology reference and glossary
• Anatomical planes
  • Sagittal: left and right
  • Coronal: front and back
  • Transverse: top and bottom

    

• Anatomical terms of location
  • Bipeds vs. quadrupeds
  • Dorsal toward belly, ventral toward back
  • Anterior toward front, posterior toward back
  • Proximal toward main mass of the body, distal away
  • Medial toward center, lateral toward sides

    

• Anatomical terms of motion
  • List of joints and their movements
  • Muscular contractions
    • Concentric resulting in its shortening, eccentric during its lengthening
    • Dynamic resulting in movement, isometric without significant movement
  • Joint motions
    • Flexion decreases joint angle, extension increases
    • Abduction moves away from center of body, adduction toward
    • Rotation: external rotates away from center of body, internal toward
    • Rotation: pronation results in upward facing, supination results in downward facing

      

      

• Anatomical structure
  • Spinal column

    

Shell

Intro

• Assuming the Bash shell
• Cheat sheet
• Howto (basics) (beginners) (advanced)
• Online playground (sh via JSLinux)
• Reference (GNU Bash reference manual)

General

• Startup scripts (reference)
  • ~/.profile for any login shell, e.g. setting environment variables
  • ~/.bashrc for interactive Bash usage, e.g. aliases
  • ~/.bash_profile for Bash-specific login shells
  • Exception: for Mac’s Terminal.app, each new terminal window runs a login shell
• Path
  • hash: common commands can be added to hash lookup, superseding normal path lookups
    • Indication: type rm outputs e.g. rm is hashed (/bin/rm)
    • hash -d vim reload hash table entries for vim
    • hash -r reload all hash table entries
• Readline: line-editing and history for Bash and other command-line interfaces (wikipedia)
  • Commands (assuming vi mode)
    • Cheat sheet (vi mode) (emacs mode)
    • ctrl-r, ctrl-s reverse/forward substring search
    • v edit current command in editor, then execute (:cq from Vim to abort)
  • Options
    • set editing-mode vi set vi mode for all Readline-dependent tools
  • Expansions
    • !* all previous aruments
    • !^ first of previous aruments
    • !$ last of previous aruments
    • !2 second of previous aruments
    • !2-3 second and third of previous aruments
• History (intro)
  • history -r read history file and append contents to history list
  • history -a append new history lines to history file
  • history -s add args to end of history list as single entry
  • History expansion (stackoverflow)
    • !:1 first argument of previous command
    • !:1-2 first and second arguments of previous command
    • !$ last argument of previous command
    • ^cat^bat run previous command with cat replaced to bat
• Redirection (cheat sheet)
  • openssl x509 -help 2>&1 | less redirect stderr to stdout
  • cat <(echo 1) redirect stdout to anonymous fifo, then pass fifo to cat as an argument
• Completion
  • Bash doesn’t automcatically handle completion for aliases (Zsh does, however)
  • complete -p git get complete’s completion handling for git
• Exit status
  • Exit code values
    • 0 indicates successful command
    • 1-255 indicates failed command, generally acts as an error code
  • echo $? print exit status of most recent command
• Kill a running process via increasingly drastic measures
  1. ctrl-c send SIGINT
  2. ctrl-\ send SIGQUIT
  3. ctrl-z send SIGSTP to suspend program
  4. kill kill the process ID from another terminal
  5. kill -9 force-kill the process ID from another terminal, without a clean abort
• Miscellanea
  • Debug scripts by toggling debug mode via set -x and set +x
  • Programmable completion (tab expansion)
  • Functions can be exported via export -f
  • pushd ., popd, dirs to cd via a stack of directories

Scripting

• set -euf -o pipefail scripting settings for safer script exits
• Heredoc syntax supports defining a separate file inline sh cat <<EOF foo bar EOF
• Variable assignment
  • VARIABLE=${1:-DEFAULTVALUE} provide default variable value
• Control flow: loops

  • for i in {0..3} ; do echo $i ; done loops 0 1 2 3

  • for i in $(seq 3) ; do echo $i ; done loops 1 2 3

  • for (( i = 0 ; i < 3 ; i++ )) ; do echo $i ; done loops 0 1 2

  • Reading lines from file ```sh while read p; do echo “$p” done <peptides.txt

    # or

    cat peptides.txt | while read line; do echo “$line” done ```

• Control flow: conditionals
  • if [[ -d tmp ]] && [[ 1 == 1 ]] ; then echo 42 ; else echo 43; fi
• Conditional expressions
  • Comparing integers via arithmetic expression or conditional expression: both are fine
  • Single vs. double square brackets: prefer double unless POSIX compatibility is strongly desired
  • (($foo < $bar)) arithmetic
  • [[ $foo -lt $bar ]] conditional
  • [[ ! 42 == 42 ]] negate conditional
  • Common expressions
    • [[ $foo == "foo" ]] string equality
    • [[ -d $foo ]] exists and is a directory
    • [[ -f $foo ]] exists and is a regular file
    • [[ -r $foo ]] exists and is readable

Snippets

• Timed spinner

# spin for the passed number of seconds, defaulting to 10 seconds.
function spin() {
  nsec=$(( ${1-10}*10 ))
  spin='-\|/'
  for (( i = 0 ; i < nsec ; i++ )) ; do
    j=$(( i % 4 ))
    printf "\r${spin:$j:1}"
    sleep .1
  done
  printf "\r \n"
}

Options

• set -o vi set vi mode just for shell
• shopt -s histappend allow parallel shells to correctly append to the history file
• stty -ixon fix forward search in some terminals

Tools

• General
  • htop interactive process viewer

    

    

• Search
  • fzf fuzzy finder (examples)
  • rg line-oriented search
  • rga seach PDFs, Office documents, and more
  • scc count lines of code (similar to cloc, loc) (also: tokei, e.g. tokei -o json | tokei-pie to generate a recurisve pie chart)
• Data wrangling
  • fx JSON explorer (also: jless)
  • visidata tabular data explorer (also: datasette)
• Files
  • ranger Vim-like file manager (also: broot)
    • ~/.config/rc.conf startup commands and key bindings
    • cw change current filename
    • S open shell in current directory
    • :bulkrename bulk-rename selection (select via shift-v)
    • :delete delete selection
    • :touch create file (:shell touch create multiple files at once)
    • :flat -1 flatten all directories under current (e.g. for easier bulk rename)
  • rclone sync files and directories to and from local or cloud storage solutions
    • Supports Dropbox, Google Drive, FTP, HTTPS (and client-side certificate validation), and more
    • rclone serve http . serve local directory and files over HTTP
  • watchman execute commands in response to file modifications (also: entr)
• Network
  • bandwhich network utilization by various categories
  • iftop top for network usage
    • iftop -i en0 view overview for interface en0
• Media
  • easytag view and edit audio file tags
  • ffmpeg audio/video processing (common commands)
    • Intro to video technology: image, video, codec, and more
    • ffmpeg -i input.wav output.mp3 convert WAV file to MP3
  • imagemagick convert, edit, compose images
    • magick test.svg test.png convet SVG image to PNG
    • magick IMG_4281.PNG IMG_4282.PNG -quality 100 golds.pdf magick multiple images into a single PDF
    • magick drivers-license-2023.pdf -quality 100 drivers-license-2023.png magick a multi-page PDF into multiple images
    • magick mogrify -monitor -format png *.HEIC convert all HEIC images to png format
  • youtube-dl download from YouTube, Soundcloud, Funimation, and others
• Miscellanea
  • export-chrome-bookmarks export Google Chrome bookmarks and history
  • fortune random messages from a database (about)
    • $(brew --prefix)/Cellar/fortune/9708/share/games/fortunes location of fortune files (for Homebrew fortune installations)
    • % meta character separating each fortune
    • <tab><tab>-- Thérèse de Lisieux attribution format (and no " for quotes)
    • strfile catholic generate catholic.dat for consumption by fortune
  • redshift adjust display color temperature according to surroundings

Miscellanea

Technical topics

• Self-learning
  • Opinionated list of classes and accompanying material (Teach Yourself CS)
  • Full degree’s worth of classes and material (Open Source Society University)
  • Great textbooks in various field (Ask HN)
  • Programming book list (danluu)
• Compilations
  • Awesome lists
  • Free tools for devs
  • Software engineering blogs
  • The best software tech talks ever, ranked
  • Best computer science papers (best paper awards)
  • Programming aphorisms (Hacker Laws)
  • Plain text tools
• Articles
  • All about product management
  • Choose boring technology
  • You do not need blockchain
• Classes and textbooks
  • Underactuated Robotics: algorithms for walking, running, swimming, flying, and manipulation
  • Introduction to algorithmic trading
• Organizational tools
  • Notion
  • Airtable
• Other resources
  • Personal automation
    • Self-hosted solutions (deploy-your-own-sass) (awesome-selfhosted)
    • Home Assistant: open-source home automation
    • Tailscale, ngrok: expose services on local networks (firewall/NAT busting)
    • ntfy.sh: send notifications to your phone
  • yEd Graph Editor: diagrams generation
  • List of startups hiring for remote work
  • List of European tech visas and work permits
  • Awesome design tools
  • Some standard document templates, e.g. pitch, vision, strategy, product requirements, etc.

Non-technical lists

• Question lists
  • List of controversial topics (ProCon.org)
  • List of questions to get to know someone
• Busy person patterns
  • Just start
  • Contiguous time blocks
  • Single-task important items
  • Put it off
  • Seek clarification
  • Batch the simple stuff
  • Task jar
  • Strike when you are hot
  • Feedback loop
  • Prioritize
  • Delegate
  • Drop unimportant tasks
  • Good enough
• How to be successful
  • Compound yourself
  • Have almost too much self-belief
  • Learn to think independently
  • Get good at “sales”
  • Make it easy to take risks
  • Focus
  • Work hard
  • Be bold
  • Be willful
  • Be hard to compete with
  • Build a network
  • You get rich by owning things
  • Be internally driven

Life hacks

• Append a plus sign to Gmail address: john+spam@gmail.com delivered to john@gmail.com, but can be filtered e.g. as if it was spam
• Force a public Wi-Fi connection page to appear
  • The “connection page” is a web page called a captive portal
  • Often making an HTTP request will allow the network to insert the captive portal
  • Debugging steps
    1. Direct a browser to http://www.example.com (or e.g. http://captive.apple.com for Macs), in Incognito mode if necessary
    2. Explore further debugging steps
• Encode Wi-Fi access point password into a QR code

Quick links

• Personal data access (Facebook) (Google)
• Update third-party permissions (Facebook) (Google) (Dropbox) (GitHub) (Spotify) (Apple)
• Manage recurring payments (PayPal)
• Tax forms (USAA) (Coinbase) (Nelnet)
• Free access to research papers via Sci-Hub (about) (where is Sci-Hub now)
• Southwest (travel funds) (low fare calendar)
• USPS change of address form
• Web calendars
  • Sports games (ESPN) (Google)
• Public torrent sites
  • List of torrent sites (list 1) (list 2)
  • Usually just hosting magnet links with useful associated metadata, rather than full torrent files
  • How magnet links work

Sell vehicle privately

• Howto in TX
  • General info
  • Overview
  • Specifics
• Listing
  • Nice pictures of the washed car
  • Extremely detailed – VIN number, trim, features, etc
  • Forms of payment accepted
  • Screencap of CARFAX and CARFAX (or Edmunds/KBB) valuation
• Desired documents
  • Service history
  • CARFAX report
• Required documents (take pictures of all items once completed)
  • Title
  • Title transfer from new state (in TX, and many other states, also functions as bill of sale)
  • Bill of sale (if the title transfer isn’t also a bill of sale)
• Point of sale
  • Ensure all personal items removed from vehicle
  • Take picture of buyer’s license, then do test drive (preferably where buyer just rides along, but permit test drive if desired)
  • (Optional) accompany buyer for an independent mechanic’s inspection (which buyer pay for)
  • Together, withdraw a certified check from the buyer’s bank
  • Perform the switchover in the following order
    • Sign documents
    • Deposit certified check (ideally using mobile app)
    • Hand over documents
    • Remove identifying/personal info from vehicle
      • Plates
      • Registration sticker
      • Toll tags
      • Final check for personal items
    • Hand over keys
  • (Optional) accompany buyer to county tax office to support title application submission
• Followups
  • Notify state of the transfer
  • Cancel/transfer auto insurance

Food

• Apple rankings
  • Honeycrisp apples (91)
  • Pink lady apples (80)
• Beef grades

  

  

Home

• Houseplant care overview
  • Care sheets for specific houseplants (reddit) (bloomscape)

Tech

• Power cables (NEMA and IEC)

  

• USB connectors

  

Other

• Useful apps and services
  • Hipcamp: book camping experiences (Airbnb for campgrounds)
• Clothing style

  

• Leather grades (reddit) (summary) (wikipedia)
  • Highest to lowest quality
    • Full-grain
    • Top-grain
    • Split-grain (e.g. suede)
    • Genuine (just a marketing term–not a real grade)
    • Bonded
  • Can determine whether substance is real leather by running a lighter across its surface (leather is mildly resistant to burning/melting)

    

• How to hang curtains

  

  

• TV dimension and resolution calculator

  

• Exploratory, descriptive, and explanatory research
  1. Exploratory initiates look into a space, sans prior background research
  2. Descriptive attempts to gain unbiased, generally numerical data on topics in the research area
  3. Explanatory attempts to explain previous descriptive research, toward the goal of a more complete understanding of the research area
• Flight dynamics

  

• SI base unit
  • meter: length
  • kilogram: mass
  • second: time
  • ampere: electric current
  • kelvin: temperature
  • candela: luminous intensity
  • mole: amount of substance

• SI derived unit

  

• Better emailing patterns

  

Grammar and style

Style

• How to write in plain English
  • Short sentences
  • Active verbs
  • Use “you” and “we”
  • Avoid nominalisations
  • Use lists
• Use adverbs sparingly

Punctuation

• Hyphen, en dash, em dash (Chicago Manual of Style)
  • Hyphen (-) connects things that are intimately related, like “toll-free” or “two-thirds” (- in LaTeX)
  • En dash (–) connects things over a range, like “May–September” or “chapters 3–5” (-- in LaTeX)
  • Em dash (—) functions more like parenthesis, separating two mildly related parts of a sentence (--- in LaTeX)

Disambiguations

• Method vs. methodology: a methodology does not define specific methods, but rather surveys and/or compares existing methods

Abbreviations

• Latin abbreviations
  • cf. (“compare with”)
  • e.g. (“for example”)
  • i.e. (“in other words”)
• Always use a comma after “i.e.” and “e.g.”
• Don’t italicize “i.e.”, “e.g.”, or “etc.”
  • Foreign words commonly accepted in English are not italicized (Quora)
  • Common Latin abbreviations are set in roman (Chicago Manual of Style)

LaTeX

Overview

• About
• Overview of common tasks
• Common errors
• Full intro
• Documentation
• Package documentation
• Full documentations listing
• Finding symbol names (Detexify)
• Word count via TeXcount: texcount -merge main.text

Intro

• Installation

  • Via MacTex: download and install

• Document generation

  • latexmk -pvc in.tex make file and open in PDF viewer (intro) (docs)
    • latexmk -c clean up temporary tex-related files
  • Alternatively, just use Overleaf

• Minimal .tex file

  \documentclass[12pt]{article}
  \usepackage[a4paper, total={6in, 8in}]{geometry}
  \begin{document}
  Hello world.
  \end{document}

Miscellanea

• LaTeX-style math rendering in a browser
  • KaTeX: fast server-side or client-side rendering 🌟
  • MathJax: full-featured client-side rendering
• Style and quirks
  • e.g.\ normal spacing don’t put wider space after a particular period
• Math mode
  • $x \text{ is equal to } 42$ normal text in math mode using the amsmath package
• \textcolor{gray}{\xmark} change text color with the xcolor package

Citations

• Intro
  • Bibliography management with BibTeX and biblatex
• Landscape of options
  • File format
    • BibTeX (.bib)
  • Interface between .bib file and LaTeX
    • BibTeX
    • Biber 🌟
  • Package for citation formatting
    • natbib
    • biblatex (getting started) (cheat sheet) 🌟
• Process

  1. Reference management software like Mendeley creates a BibTeX file (.bib)

  2. Reference the relevant BibTeX file; e.g. by creating a symlink in the local directory

  3. Cite using the biblatex package

     % Minimal example using biblatex
     \usepackage[sorting=none]{biblatex}
     \addbibresource{bibtex-file.bib}
     % ...
     \cite{AuthorYear}
     \cite[pre-text][post-text]{Author1Year1,Author2Year2}
     % ...
     \printbibliography{}
     % ...

• Choosing a reference manager
  • Favorite options
    • Mendeley 🌟
    • Zotero
    • Papers
  • Comparison
    • Mendeley vs. Zotero (Quora) (Hacker News) (York University)
    • All options

General

• Tikz (intro) (docs)

  • Examples
    • Tree

    \begin{figure}[ht]
    \centering
    \begin{tikzpicture}[
        grow = left,
        sibling distance = 2em,
        level distance = 8em,
    ]
    \node {payload}
        child {node {TLS}
            child {node {HTTP} edge from parent[<-,black]}
            child {node {IMAP/POP3} edge from parent[<-,black]}
            child {node {arbitrary application} edge from parent[<-,black]}
            edge from parent [<-,red]
        }
    ;
    \end{tikzpicture}
    \caption{Reuse of TLS as the mechanism for secure communication.}
    \label{fig:tls-reuse}
    \end{figure}

• Including a personal macros file

  • Include in the preamble with \usepackage{hcg-personal}

  % hcg-personal.sty
  \NeedsTeXFormat{LaTeX2e}
  \ProvidesPackage{hcg-personal}[2019/04/16 hcgatewood's personal style file]
  
  % Be sure to use \RequirePackage instead of \usepackage
  \RequirePackage{amssymb}
  
  % ...

• Hyperlinks

  • Use the hyperref package, and always load it last (docs)
  • \href{url}{text} link to url
  • \url{url} link to url, without the representative link text
  • \hyperref[label_name]{''link text''} link to label_name

• Figures

  % Include a centered picture of a gull, with caption
  \begin{figure}[ht]
  \centering
  \includegraphics[width=0.5\textwidth]{gull}
  \caption{A picture of a gull.}
  \label{fig:gull}
  \end{figure}

• Tables

  \begin{table}[ht]
  \centering
  \begin{tabular}{c|c|c|c}
  \multicolumn{1}{c}{} & \multicolumn{1}{c}{\underline{Operator co-located}} & \\
  & Agent & Controller & Witness \\
  \hline \hline
  Manned & \cmark & \cmark & \cmark \\
  \hline
  Unmanned & & \cmark / \xmark & \cmark \\
  \hline
  Logical swarm & & \cmark / \xmark & \cmark \\
  \hline
  Swarm constituents & & & \\
  \end{tabular}
  \caption{Comparing manned, unmanned, and swarm systems.}
  \label{tbl:manned-unmanned-comparison}
  \end{table}

• Gantt charts

  • With pgfgantt

    % Example calendar-based Gantt chart
    \ganttset{calendar week text={\currentweek}}
    \begin{ganttchart}[
        inline,
        x unit=0.7mm,
        time slot format=isodate,
    ]{2018-12-03}{2019-05-24}
      \gantttitlecalendar{year, month, week} \\
      \ganttbar{Synthesize}{2018-12-15}{2019-02-01}
    \end{ganttchart}

• Forest, for better trees

  • Example

    \begin{figure}[ht]
    \centering
    \begin{forest}
        for tree = {minimum size=3em},
        internal/.style = {draw,circle},
        affected/.style = {internal,fill=lightgray},
        changed/.style = {internal,dashed,edge={dashed},for children={edge={dashed}}},
        leaf/.style = {draw,square,fill=black,text=white,font=\sffamily\bfseries},
        [k,affected
            [k14,affected
                [k12,internal
                    [k1,internal [u1,leaf]]
                    [k2,internal [u2,leaf]]
                ]
                [k34,affected
                    [k3,changed [u3,leaf]]
                    [k4,internal [u4,leaf]]
                ]
            ]
            [k58,internal
                [k56,internal
                    [k5,internal [u5,leaf]]
                    [k6,internal [u6,leaf]]
                ]
                [k78,internal
                    [k7,internal [u7,leaf]]
                    [k8,internal [u8,leaf]]
                ]
            ]
        ]
    \end{forest}

Machine learning

• Tools
  • fastText efficient text classification and representation learning (tutorial)
• Resources
  • PracticalAI practical approach to machine learning
  • Neural networks in JavaScript
• Articles
  • Lessons learned building an ML trading system that turned $5k into $200k

Distributed systems

• Consistency (wikipedia)
  • Simplified overview of consistency
  • Sequential consistency vs. serializability: sequential consistency is unconcerned with interleavings or the possibility of interleavings, as long as each client’s operations are enacted in the order specified by the client
  • Sequential consistency vs. linearizability: many possible/legal interleavings under sequential consistency, while linearizability forces a particular interleaving based on global timestamps
  • Consistency models

    

• Papers
  • MapReduce paper
  • Raft paper
  • Raft state machine

    

Algorithms

Resources

• Combinatorial optimization suite (Google OR-Tools)

Algorithms

• Linear-time median finding
• Reservoir sampling: choosing k items uniformly and efficiently from a list of n items, where n very large
• Dynamic programming
  • Seam carving: content-aware image resizing (intro) (improved)

Data structures

• Visualizations
• Bloom filter space-efficient probabilistic data structure reporting set membership (“no” or “probably”) (visualization)
  • Cuckoo filter same function, often better in practice (visualization)
• How to implement strings
• Hexagonal grids

  

• Rope: store and manipulate a very long string, e.g. by a text editor

  

• Wave function collapse: take in an “archetypal” input, then use that implicit set of rules to produce procedurally-generated output; generall used for images or 3D designs

  

Probability

• Prior probability vs. posterior probability
  • Prior is without additional evidence
  • Posterior is including evidence
• Moments of a random variable:
  • 1st moment: mean
  • 2nd central moment: variance
  • 3rd central moment: skewness
  • 4th central moment: kurtosis
• Making sense of standard deviation
• Intuitive understanding of covariance

  

• Poisson distribution

  

• Exponential distribution (the only memoryless distribution)

  

• Beta distribution: represents prior belief about a particular probability; allows recursive updating

  

• Convolutions
• Logarithm rules

  

Wolfram Alpha

• P(n, k) k-permutations from n-sized set
• C(n, k) k-combinations from n-sized set
• k+1 where k=2 evaluate for some assigned variable

Pandoc

• User guide
• Markdown
  • GitHub markdown emoji (full list) (rendered) (sorted subset)
• Markdown to HTML
  • Download a CSS style as pandoc.css
  • pandoc --standalone --self-contained --css=pandoc.css --katex=katex_0.10.0/ --from markdown+smart+emoji in.md -o out.html convert in.md to standalone out.html
  • Can upload directly to GitHub Gists and view the HTML via GitHub HTML Preview
• Markdown to PDF
  • pandoc --from markdown+smart --pdf-engine=xelatex -V urlcolor=blue in.md -o out.pdf
  • pandoc --from markdown+smart --pdf-engine=xelatex --template eisvogel --listings in.md -o out.pdf using a template

Future topics

• 3D printing
  • Onshape: 3D printing design tool
• Software-defined radio (SDR)
  • Intro

Raspberry Pi

Intro

• Documentation: setup, installation, usage, etc.
• Command cheat sheet
• Awesome Raspberry Pi
• Serial howto

Boards

• Raspberry Pi
  • Raspberry Pi 3 Model B+ $35 board
  • Raspberry Pi Zero W $10 board with built-in Wi-Fi and Bluetooth
• NanoPi
  • NanoPi NEO Plus2 $35 board, similar to Raspberry Pi 3 Model B+
  • NanoPi Duo $15 board with Wi-Fi

Physical components

• Pinout reference (and for other related boards)
• Power from GPIO pins: 5V to 5V, GND to GND, but no regulation/fuse protection against over-voltage and current spikes

Install Raspbian

1. Download Raspbian
2. Unzip and copy image to SD card
   • Copy image to SD: dd if=2018-04-18-raspbian-stretch.img of=/dev/sdX bs=4M conv=fsync status=progress
   • Single step: unzip -p 2018-04-18-raspbian-stretch.zip | dd of=/dev/sdX bs=4M conv=fsync status=progress
   • Can also use cat instead of dd
   • On some machines, had an issue with accessing the unzipped file–so just use the single step command
3. Safely remove SD card: sudo sync
4. Insert SD card into Pi, power on, plug in HDMI monitor and USB peripherals, boot
5. Edit desired config values in /boot/config.txt and reboot

Software

• picamera: Python bindings for Raspberry Pi camera module
• pigpio: circumvent requiring root for access to GPIO pins requires root
• WiringPi: GPIO access, including hardware PWM
  • For NanoPi boards, use WiringNP
  • gpio command-line utility (requires root)
    • gpio readall view (somewhat idiosyncratic) pin number mappings
    • gpio mode 1 pwm set pin 1 to use PWM
    • gpio pwm 1 1023 set pin 1 to max PWM value
    • gpio mode 16 out set pin 16 to output mode
    • gpio write 16 1 set pin 16 to high
  • Python bindings

Fun projects

• General guides (intro) (learning) (all)
• “Super computing” cluster

Peripherals

• USB-to-UART adapter: Adafruit USB-to-TTL serial cable

Hardware

• Raspberry Pi UARTs: disabled by default
• RS-232 vs. TTL Serial (UART)
• I2C: Inter-Integrated Circuit (intro) (in-depth)
• Power supply: CV means “constant voltage”, CC means “constant current”
• Level shifter vs. voltage regulator: level shifter converts one digital signal to another, voltage regulator produces constant output voltage from higher, mildly variable input voltage
• How DC motors work, part of the intro to motors series

Miscellanea

• Micro Center store in Boston
• Assign static IP address: edit /etc/dhcpcd.conf then reboot
• Armbian is a Debian-based Linux distribution (documentation)
• Open window options, e.g. to resize: <alt><space>
• Raspberry Pi-specific system information with vcgencmd (video core general commands): vcgencmd commands
• Playing video requires hardware acceleration, provided e.g. by OMXPlayer (part of default Raspbian installation)
• Yellow lightning bolt icon means under-voltage warning: try different USB power cable

Linux

General

• Linux sandbox in the browser (JSLinux)
• Build custom Linux system from existing source code (Linux from Scratch)

Debian

• General information, including network services
• Official documentation
• Systemd: a system and service manager, among other features

Ubuntu

• Official documentation
• All man pages, across versions
• Get Ubuntu version: lsb_release -a
• Change hostname persistently: edit /etc/hostname
• Package management
  • apt list --installed list installed packages
  • Installing a newer version of Python
  • Source packaging is required for Debian packages
  • Supported hardware: i386, amd64, armel, and more
    • i386 vs. amd64: 32-bit vs. 64-bit of x86 architecture
• Snaps: self-contained, transactionally-updated method for software distribution targeting IoT devices

Performance

• Linux debugging tools intro
• Performance tools and overview (Brendan Gregg)

  

Networking

• Routing and traffic control howto
• Configuring network interfaces
• Interface configuration for ifup and ifdown
• Parse host’s local IP address
  • hostname -I list “all” resolvable addresses for the host
  • ip -f inet addr show eth0 | grep -oP 'inet \K[\d.]+' get the IPv4 address on interface eth0 (\K means “output starting here”)
  • Other options (SO 1) (SO 2)

Proxies

• Set proxy location: something like the following in /etc/environment (username:password only as-necessary):

  export http_proxy=http://username:password@proxyhost:port/
  export https_proxy=https://username:password@proxyhost:port/
  export ftp_proxy=http://username:password@proxyhost:port/

• Run sudo commands through a proxy: add line Defaults env_keep = "http_proxy ftp_proxy" using visudo command

File system

• Partition: logical section of a physical disk acting as a unique disk
• Block devices and block sizes
  • Block size used by file system
  • Sector size used by disk/storage
  • Block size may be a multiple of sector size
  • Block device: hardware component storing blocks of bytes (e.g. hard disk rather than keyboard)
• blockdev
  • blockdev --getbsz /dev/sda get block size of sda
  • blockdev --getsize64 /dev/sda get total size of sda
• dd
  • Can be used to create/restore/clone/transfer, rescue file with bad blocks, benchmark device throughput
• fdisk
  • fdisk -l | grep "Sector size" disk sector size
• stat
  • stat file information about file
  • stat -f file information about filesystem on which file resides
  • Three types of timestamps
    • atime: access, or last time file was read
    • mtime: modify, or last time file content was modified
    • ctime: change, or last time file metadata was modified
  • Understanding timestamps
    • Use touch to update timestamps
    • atime is generally not straightforward, often replaced with relatime
• Permissions

  

systemd

• Intro
  • Cheat sheet
  • Debian documentation
  • Ubuntu documentation
• Snippets
  • sudo journalctl print all logs
  • sudo journalctl -f follow logs tail
  • sudo journalctl -u jcaus print all logs for jcaus unit
  • sudo systemctl enable ~/some.service enable from unit file at arbitrary location
• Unit file
  • Assign unit an alias: alias=jcaus.service in the [Install] section
  • Minimal example, starts at boot time

    1. Create unit file

       # /etc/systemd/system/big-old-test.service
       [Unit]
       Description=a big old test
       
       [Service]
       ExecStart=/bin/bash -c "while true; do date; sleep 1; done"
       
       [Install]
       WantedBy=multi-user.target

    2. Reload systemd: sudo systemctl daemon-reload

    3. Enable service: sudo systemctl enable big-old-test

    4. Optionally, start right away: sudo systemctl start big-old-test

Commands

• Overview and list
• dmesg read/write kernel messages, e.g. to check Ethernet cable connection and status
• lsblk list block devices (type of device file), where e.g. mmc refers to a MultiMediaCard
• lscpu display information about the CPU architecture
• lsusb list USB devices
• mmv move and rename files in bulk
• passwd change password
• ping send ICMP ECHO_REQUEST to network hosts (ping6) (ping vs. ping6)
• pwgen generate passwords
• ssh secure login and shell
• strace trace system calls and signals (cheat sheet)
• sysctl view and configure kernel parameters at runtime
• tar save many files as a single file
• time time a simple command
• uname print system information like OS, kernel name, machine hardware name
• vimdiff view useful diffs between two files (cheat sheet)

Snippets

• #!/usr/bin/env bash most portable hashbang
• command gist twilio.js call non-aliased version of gist
• cp foo{,bar} expands to cp foo foobar
• date +%s number of seconds since the Epoch
• date --set='Fri Nov 22 23:36:40 UTC 2019' set system time
• date -u date and time in UTC
• dhclient eth0 configure interface eth0 with dynamically-assigned IP
• dmesg -w like tail -f for dmesg
• du -s . | awk '{print $1}' get recursive size of current directory in bytes
• echo -e 'hello\nworld' print literal newline (can also use printf)
• echo foobar | gzip | zcat pipe directly to gzip
• env print environment variables
• find . -exec touch {} \; recursively change date on all files in directory
• find . -type f -name '*.go' -print0 | xargs -0 sed -i '' 's/MapMany/Map/g' recursively replace in all files
• grep -RI foobar . recursively search all text-like files for foobar
• gsed 's/\x1b\[[0-9;]*m//g' < terraform_apply.txt > terraform_apply.log remove ANSI color codes from text
• hg status | sed -n 2p | awk '{print $2}' get line 2, column 2 of hg status output
• ip -f inet addr show eth0 | grep -oP 'inet \K[\d.]+'
• man 2 unshare view the man page for unshare from section 2
• mdfind -name .txt -onlyin ~/magma/ macOS version of find using Spotlight index
• netstat -tulnp list ports and determine what process is bound to a port
• nmap -p - localhost localhost port scan
• nohup livedown start notebooks_main.md --port 8080 & start background command to last past terminal session
• openssl rand -hex 20 generate random password
• pbpaste | jq pretty-print copied JSON (replace pbpaste with appropriate clipboard manager)
• ps -o ppid= 50377 get parent PID of PID 50377
• ps aux | grep -i net case-insensitive search for processes matching net
• rename 's/foo/bar/' \* rename files by replacing all occurences of foo with bar
• route -n show routing information like gateway and genmask
• sleep 4 sleep for 4 seconds, or use m, h, d for minutes, hours, days
• sort favorite_words.txt | uniq -d print non-unique lines
• sudo -E python3 preserve environment variables when sudo-ing
• sudo hwclock --set --date 'Fri Nov 22 23:36:40 UTC 2019' && sudo hwclock --hctosys update hardware clock
• sudo ncdu -x -o out.ncdu / && ncdu -f out.ncdu device disk utilization TUI
• tail -f out.log | ag 'foo|bar' --passthru highlight streamed log file for foo and bar (case-insensitive)
• tail -f out.log | egrep 'foo|$' -i --color highlight streamed log file for foo with grep (case-insensitive)
• tar czf target.tar.gz file1 file2 file3 create gzipped tar file
• tar xzf source.tar.gz extract gzipped tar file
• textutil -convert txt main_tf_08_06_2020.rtf convert to/from plaintext
• unset COMPOSE_FILE unset environment variables
• watch ip addr watch and wait for IP changes
• xxd -p < /dev/input/event1 convert binary data from event1 to printable format

Miscellanea

• Namespaces isolate access to global system resources (overview) (man)
• Network interfaces and network namespaces
• I/O buffering
  • Buffering rules for stdout
    • stdin: line-buffered
    • stdout (TTY): line-buffered
    • stdout (not a TTY): fully-buffered
    • stderr: unbuffered
  • Change a stream’s default buffering mode programmatically using setvbuf
  • Force stdout to be line-buffered:
    • Programmatically: setlinebuf(stdout);
    • From shell using stdbuf: stdbuf -o L grep jcaus /var/log/syslog
• Set tab completion for shell alias:
  1. complete | grep systemctl
  2. complete -F _systemctl sc
• Point-to-point connection over Ethernet (on Ubuntu): assign static, optionally private IP addresses to each machine
• Sysbench: benchmark OS parameters
• Pseudo file system: on Linux /proc holds process-specific information, /sys exports kernel objects
• Install VMWare Fusion with Ubuntu on a Mac
• Read device events (from e.g. USB devices) with evdev using libevdev (Python bindings with the evdev module)
• gzip vs. zip: tar is archiver, gzip is compressor; zip does both in one step
• Tarball: collection of files archived as a single file (from Tape ARchive)
• Locks
  • Semaphore vs. lock/mutex: only the thread that locked a mutex is supposed to unlock it
  • Mutex vs. spinlock: mutex puts thread to sleep when not available, while spinlocks have thread keep retrying

Version control

General

• Copybara transforming and moving code between repositories

Git

• General
  • Common commands
  • In-depth
  • Undo
    • How to undo almost anything
    • How to get out of a scary situation (Flight rules for Git)
  • Implement your own version of Git
• Tools
  • diff-so-fancy cleaner diffs
    • ^(Date|added|deleted|modified|renamed): search term to jump between changed files
  • gh official GitHub CLI
  • git-quick-stats generate statistics, e.g. _GIT_BRANCH=master git-quick-stats --git-stats-by-branch
  • hub GitHub wrapper
  • scm_breeze easier Git file references
  • tig TUI
• Commands
  • Reset
    1. Move the branch HEAD points to (--soft)
    2. Make the index look like HEAD (default)
    3. Make the working directory look like the index (--hard)
  • Diffs
    • git diff remotes/origin/master --name-only only names of changed files
  • Rewriting history
    • git commit --amend rebase/replace most recent commit (and message)
    • git rebase -i HEAD~3 interactive rebase of most recent 3 commits (HEAD~3 actually points to the parent of the last commit to update)
    • git push origin my-branch -f force push to my-branch
  • Branches
    • git branch -a list all branches (local and remote)
    • git branch -d local-branch delete local-branch
    • git checkout --track remotes/origin/remote-branch checkout and track remote-branch
    • git merge other-branch merge other-branch into current branch
    • git checkout -b new-branch a9c146a create new-branch from commit
  • Searching
    • git log --oneline -S hosting-helm-private commits which changed the occurrences of the string
• GitHub
  • GitHub Actions workflow events, e.g. pull_request, pull_request_target, etc.
  • git-history view file changes across commits
  • Chrome extensions
    • refined-github overall better GitHub experience
    • notifier-for-github desktop notifications

Web programming

Chrome DevTools

• Console utilities
  • $('#header') return first element with ID header
  • $$('img') return all img elements
• Snippets
  • $$('.check').forEach((e) => e.click()) click all elements with class check

NPM

• Docs
• Change NPM global install location
• Snippets
  • npm install -g git+https://github.com/hcgatewood/livedown install (global) package directly from GitHub

DOM

• Edit
  • Vanilla JavaScript
  • JQuery
  • React

JavaScript

• Chart.js: HTML5 charts

REST

• Microsoft REST API guidelines

HTTP

• HTTP/2 uses a binary protocol to multiplex multiple HTTP requests over the same TCP connection (intro) (spec)
• URLs
  • Specification (RFC 3986)
  • URI := scheme:[//authority]path[?query][#fragment]
  • authority := [userinfo@]host[:port]
  • URI, URL, URN: URL identifies a resource by its (current) location, URN uniquely identifies a resource by its name, and URI is the union of the URL and URN types
• How tracking pixels work
• HTTP caching
  • Overview
  • Disable browser caching (Chrome) (Safari)
  • Shared vs. private caching

    

  • Cache-Control header

    

• Page caching (bfcache)
  • Different form of caching than HTTP caching
  • Detect via pageshow and pagehide window events
  • Disabling the page cache

  // Reload the window when page loaded from page cache
  pageshow = function (event) {
    socket.emit('ready')
    if (event.persisted)
      window.location.reload()
  }
  
  // Hide the body when leaving page to prevent
  // "blinking" on navigation back/forward to page
  pagehide = function (event) {
    // The '$' operator can be from e.g. zepto.js
    $('body').css('opacity', '0')
  }
  
  // Install listeners
  // - pageshow: any time the page is loaded, whether from
  //      page cache or otherwise
  // - beforeunload: any time the page is destructed, whether
  //      from forward/back or closing
  window.addEventListener('pageshow', pageshow)
  // window.addEventListener('pagehide', pagehide)
  window.addEventListener('beforeunload', pagehide)

Operating systems

Reference

• MIT 6.828: Operating System Engineering
• OSDev: operating systems wiki
• Intel 80386 Reference Programmer’s Manual: manual for the original i386 microprocessor, the earliest in the x86 family
• Writing an OS in Rust (1st edition) (2nd edition)
• Linux kernel in-depth (A Heavily Commented Linux Kernel Source Code)
  • Interactive map of Linux kernel
• What every programmer should know about memory
• Example OS’s
  • xv6: didactic, written in C for x86
  • Redox: written in Rust for x86
  • TNeo: RTOS, written in C for ARM microcontrollers (overview)
  • Linux: written in C, cross-platform (topics)
• Assembly
  • Hello world in Linux x86-64 assembly
  • Cheat sheet (x86-64 reference sheet - GNU assembler format)
  • Overview (A fundamental introduction to x86 assembly programming)
  • Intro (x64 cheat sheet)
  • Wikibook (x86 assembly)
  • Extended intro (PC assembly language)
  • Understanding assembly language
  • GCC inline assembly
• Libraries
  • lwIP: lightweight TCP/IP stack
• Virtualization
  • Overview (from CMU) (from Harvard)
  • QEMU docs
  • QEMU networking
• Hardware
  • Overview (Modern microprocessors - a 90-minute guide)
  • Intel’s E1000 data sheet
• Collection of mostly low-level topics
• Patterns
  • Read-copy-update (RCU): use atomic pointer reads/writes to enforce coherent data structure views, and many-reader/single-writer synchronization to determine when an old object can be freed

General

• Memory addressing

  mov (%ebx), %eax         /* load 4 bytes from memory address in ebx into eax */
  mov %ebx, var(,1)        /* move contents of ebx into the 4 bytes at memory address var */
  mov -4(%esi), %eax       /* move 4 bytes at memory address esi+(-4) into eax */
  mov %cl, (%esi,%eax,1)   /* move contents of cl into the byte at address esi+eax */
  mov (%esi,%ebx,4), %edx  /* move the 4 bytes of data at address esi+4*ebx into edx */

• Trap vs. interrupt: trap is an interrupt generated in a user process (software-generated exception); an interrupt in general can be asynchronously generated by hardware

• PHY: circuitry connecting the link layer device (e.g. a wider Ethernet NIC) to the physical medium

• Memory-mapped vs. port-mapped I/O

• CMOS, aka nonvolatile BIOS memory: stores e.g. system time, BIOS settings

• Recursive page tables (in xv6) (general description)

• Virtual to physical address translation:

  

• Translation registers:

  

• Interrupt description table:

  

• Stack during subroutine call:

  

• Inode (disk inode in xv6):

  

• Network server environments in JOS (we implemented green boxes):

  

• Processor vs. CPU:

  

Golang

Overview

• Intro
  • Tour of Go
  • Writing, building, installing, and testing Go code
• Reference
  • Tutorial (video)
  • Reference (Effective Go)
  • Examples (Go by example)
  • Specification (Go programming language specification)
  • In-depth experiences (Experience reports)
  • Style
    • Advice (Practical Go)
    • Google style guide (Go code review comments)
      • Error strings lowercased
      • Initialisms consistently-cased, e.g. ID rather than Id
      • Interface generality: if a type exists only to implement an interface, the type should be private and a constructor function returns the interface
      • Delcaring empty slices
  • Generated code
    • A Go file is assumed to be generated if any of the lines before the first non-comment, non-blank line match the regex ^// Code generated .* DO NOT EDIT\.$
  • Testing
    • Testing patterns
    • stress runs a test until it fails (install with go get golang.org/x/tools/cmd/stress)
• Miscellanea
  • Go is object-oriented: interface implementation and embedding provide subtyping (is-a) and object composition (has-a) relationships between structs and interfaces
  • Nil-safe getter for nested structs, as in generated protobuf code, checks if the receiving struct pointer is nil
  • Prefer &Obj{} over new(Obj) where possible, as the struct literal is newer and idiomatic for applicable use-cases
  • Different types of nil (Understanding Nil)
  • // Deprecated: use alternative... mark object as deprecated
• Standard libraries
  • context cross-boundary context cancelation (intro)
  • encoding/json
  • fmt formatted I/O, including print
    • %v default format, e.g. for structs
    • %+v add field names for structs
    • %#v print in full Go syntax
    • %T type of value
  • net/rpc
  • strconv string conversions
  • strings
• Performance
  • Debugging performance issues in Go programs
• Concurrency
  • Patterns
  • Data race detector
  • Channel implementation
  • How goroutines work
  • How to combine goroutines with sync.WaitGroup
  • Communicating sequential processes (Bell Labs and CSP Threads): inspiration for Go’s concurrency model
• Memory
  • Pointer arithmetic not allowed
  • Memory allocation
  • Nil
• Strings
  • Strings, bytes, runes, and characters
  • Back quotes (backticks: `) encase non-escaped string literals
• JSON
  • Overview
  • Streaming encoders/decoders
    • Wrapper around io.Reader, io.Writer
    • enc := json.NewEncoder(<file>) create encoder
    • err := enc.Encode(&v) encode v to file
• Testing
  • Inter-package tests are run concurrently by go test, by default
• Tools
  • panicparse parse panic stacks into human-readable text
• Snippets
  • go test -count=1 ./services/state/... force no-cache test run
  • import assert "github.com/stretchr/testify/require" have failed assertions end the test immediately
  • go test -c ./services/lte/obsidian/handlers/ && while ./handlers.test -test.run=TestAPNResource -test.count=100 -test.failfast ; do date ; done run flaky test till failure

Mac

Specific apps

• Terminal
  • Install Solarized theme by downloading Solarized and installing new profile in Terminal options
• Apple Calendar
  • View all events in specific calendar
    1. Search .
    2. Deselect all other calendars
• Spotify
  • Fix ineffective “forget device” (reset Spotify Connect device list)
    1. Log in to Spotify
    2. Go to “apps” page, remove access for Alexa
    3. Go to “account overview” page, click “sign out everywhere”
    4. On mobile, go to Alexa app > Settings > Music & Podcasts > Spotify, disable and re-enable Spotify skill
  • Fix cached local files issue
    1. Close Spotify
    2. Remove ~/Library/Application Support/Spotify/Users/hcgatewood-user/local-files.bnk
    3. Reinstall Spotify

General

• Peripheral ports
  • USB-C docks (Show HN)
  • Overview (Apple)
  • Thunderbolt 3 vs. USB-C: basically the same, Thunderbolt 3 is a superset
  • Thunderbolt 2 vs. Mini DisplayPort: Thunderbolt 2 is a superset
• Options
  • Set the primary display on multi-display setup by moving the white bar in system preferences > displays > arrangement
  • sudo mdutil -a -i off disable Spotlight indexing
• Understanding common processes found in the Activity Monitor
• File system
  • ~/Library/Application Support user-specific application storage
• Troubleshoot issues with AirPods or other Bluetooth devices
  1. Forget and re-pair the device
  2. Update computer’s operating system 🌟
  3. Reset Bluetooth module then restart computer: shift-alt-click Bluetooth icon > Debug > Reset Bluetooth module
  4. Factory reset AirPods: hold down AirPods case button for 15 seconds
  5. Try other potential solutions
• Markdown
  • markdown-toc --maxdepth 2 notebooks_main.md | pbcopy generate the table of contents for this file using markdown-toc
• Convert text to speech stored as an audio file
  • Copy the desired text (or save to text file)
  • pbpaste | say -o out.aiff convert to speech stored as AIFF format
    • pbpaste pastes from the system clipboard
    • -r 250 set output to speak 250 words per minute
    • -f in.txt rather than taking from the clipboard, pull from in.txt
  • Open out.aiff using e.g. VLC
    • VLC allows strong control over playback speed
• Run a service at login

  • Using launchctl (overview) (docs)

  • Example launchctl plist file

    <!-- ~/Library/LaunchAgents/com.user.login.plist -->
    <?xml version="1.0" encoding="UTF-8"?>
    <!DOCTYPE plist PUBLIC "-//Apple Computer//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
    <plist version="1.0">
    <dict>
        <key>Label</key>
        <string>com.user.login</string>
        <key>ProgramArguments</key>
        <array><string>/Users/Hunter/.login.sh</string></array>
        <!-- <key>StandardOutPath</key> -->
        <!-- <string>/Users/Hunter/.login.out</string> -->
        <!-- <key>StandardErrorPath</key> -->
        <!-- <string>/Users/Hunter/.login.out</string> -->
        <key>RunAtLoad</key>
        <true/>
    </dict>
    </plist>

  • Example login script

    # ~/.login.sh
    PATH=/usr/local/bin:/usr/bin:/bin
    livedown start '/Users/Hunter/Dropbox (Personal)/notes/notebooks_main.md' --port 8081

Keyboard shortcuts

• Intro
  • Remap any keys, e.g. with Karabiner-Elements
  • Remap switch control and caps lock keys
• General
  • Multi-touch gestures
  • cmd-ctrl-q lock mac
  • ctrl-down app expose
  • ctrl-up mission control
  • fn-up page up
  • fn-down page down
  • cmd-b open Spotlight contents as a search engine query
  • cmd-ctrl-space emoji search
• Finder
  • cmd-shift-. show hidden files in Finder
  • cmd-shift-g go to folder, with tab complete
• Terminal
  • alt-click move cursor location to mouse pointer
  • cmd-2click open URL
  • cmd-shift-2click select URL or file path
• Screenshots
  • cmd-shift-3 screenshot whole desktop
  • cmd-shift-4 screenshot specified region
    • Before selection
      • space specific window
    • During selection
      • space hold to move chosen area
  • cmd-shift-5 screenshot options and recordings
• Chrome
  • cmd-shift-t undo closing tab
• YouTube
  • , back one frame
  • . forward one frame
• Toggle connection to Bluetooth device (also consider ToothFairy)
  • BluetoothConnector to wrap Bluetooth device control
  • BetterTouchTool to run BluetoothConnector as a shell script
  • /usr/local/bin/BluetoothConnector 00-00-00-00-00-00 --notify, substituting desired device’s MAC address
• Vimium
  • Commands
    • ? help, with option to show all/advanced commands
    • yy yank current URL
    • P paste clipboard in omnibox of new tab
    • gu go up URL hierarchy
    • gU go to root of URL hierarchy
    • gi focus first text input on the page
    • yf copy link URL
    • yt duplicate current tab
    • W move current tab to new window
    • >>, << move tab to the right/left
    • ]], [[ go to next/previous page (e.g. for Google search results)
  • Visual mode
    1. Use find mode to select starting point
    2. c to enter caret mode (optional)
    3. v to enter visual mode
    4. y, Y to yank selected text
  • Preferred mappings map H previousTab map L nextTab map J goForward map K goBack map ' Marks.activateGotoMode map \ passNextKey normal

Homebrew

• Documentation
• Related: mas, a CLI for the Mac App Store
• Snippets
  • brew update-reset hard update to fix any potential merge issues
  • brew cleanup remove all old formulas
  • brew update && brew upgrade update Homebrew, then upgrade all outdated packages
  • brew deps --installed list top-level formulas (seems to not be working…)
• Debugging
  • z will sometimes stop working and cause abort trap: 6, fix with brew reinstall gawk

Essential apps

• Other lists (Nikita Voloboev) (Awesome Mac)
• Built-in
  • Digital Color Meter
• Amphetamine: prevent entering sleep mode
• BetterTouchTool: keyboard shortcuts to move and resize windows, run shell scripts, etc.
• Cleanup.pictures: fantastic tool to quickly clean up pictures
• Codeshare: free, shared online codepad (alternative: Collabedit)
• 🌟 Excalidraw: free online sketching and diagramming, with collaboration
• Flow: Pomodoro timer to automate work/break cycles (keyboard shortcuts)
• F.lux: adjust display color temperature depending on location and time of day
• Flycut: clipboard manager to store and paste up to 99 stored snippets at a time
• HackMD collaborative Markdown, and can use a GitHub repo as backing store
• Howler Pro: timer with notifications to set and record sprints through the day
• NoPaste: online serverless pastebin
• 🌟 Obsidian: Markdown-based note taking app with Vim-mode, sync, and publish features
• Read Aloud: browser extension to convert web page text to spoken audio
• SequenceDiagram.org easy, shareable sequence diagrams
• Vimium: Vim-like shortcuts in Google Chrome

Python

Intro

• Overivew (The Hitchhiker’s Guide to Python)
• Cheat sheet (gto76)
• Cheat sheet and patterns (pysheeet)
• Python howtos

Installs

• pyenv: switch between versions of Python, locally or globally, by injecting shims into the PATH (commands)
  • pyenv version currently set Python version
  • pyenv versions all installed, pyenv-selectable versions
  • pyenv install --list list Python versions available for installation
  • pyenv install 3.4.10 install version as a pyenv-selectable version
  • pyenv local show version local to current working directory
  • pyenv local 3.4.10 set local version for current working directory
  • pyenv local --unset unset local version
  • pyenv global 3.7.4 set global (per-user) version
• Install pip
  • Overview
  • Pip on Ubuntu: sudo apt-get install python-pip python-dev build-essential
  • Pip3 on Ubuntu: sudo apt-get install python3-pip
• Install IPython: python3 -m pip install ipython
• Install Python modules: python3 -m pip install some-package

Style

• Avoid Hungarian notation, i.e. don’t include type information in an identifier’s name

General

• Related tools
  • MicroPython lean Python implementation targeting microcontrollers and constrained environments
• Language miscellanea
  • Docstring format python def some_function(): """A default one-line docstring.""" pass
  • Unpacking (splat) operator
  • Floats
    • m, e = math.frexp(4.2); m * 2**e == 4.2 exactly (extract float’s mantissa and exponent)
    • a, b = (4.2).as_integer_ratio; 4.2 == a / b exactly
  • String formatting
    • '{:5.5}'.format('xylophone') truncates and pads to exactly xylop
  • open modes
    • r read
    • w write (truncate first)
    • a append
    • r+ read/write
    • b binary (don’t decode file’s bytes contents to str)
  • Async
    • await asyncio.gather(f(), g(), h()) run function calls concurrently
    • async for elt in awaitable_iterable iterate over async iterable
  • Dotted notation for dictionaries
    • Via SimpleNamespace for the most simple needs
    • Via namedtuple under limited circumstances
  • Type annotations (cheat sheet)

    • Prefer base classes where applicable

      • Iterable for generic iterables (anything usable in for)
      • Sequence where a sequence (supporting len and __getitem__) is required

    • Support for generics

    • Example ```python from typing import Callable, Union, Optional, List

      def f(num1: int, my_float: float = 3.5) -> float: return num1 + my_float

      x: Callable[[int, float], float] = f

      # Optional variables are allowed to be None def send_email( address: Union[str, List[str]], sender: str, cc: Optional[List[str]], bcc: Optional[List[str]], subject=’’, body: Optional[List[str]] = None ) -> bool: ```

• Modules
  • Multiple imports of the same module will only execute module-level code a single time
  • The __all__ = [] construct determines what happens on import * from
• Python bytecode: Python code compiled to bytecode for execution by the Python interpreter
  • Cached as .pyc files (structure) (create)
  • Interpreting the output of dis.dis(func)
  • Patching bytecode (not production-recommended)
• Socket programming howto
• Strftime reference
  • a abbreviated weekday
  • W week number of the year
  • -m month as decimal number
  • -d day of the month as decimal number
  • X time, with seconds
• Logging cookbook
  • Serializing log records from multiple processes
    • Recommended method is setting up a single process as a logging server
    • Clients make requests to the server, which then acts as the record serializer
    • Note that this socket-based approach is not thread-safe on the client’s side (and zerorpc doesn’t allow cross-thread sharing at all, so a different client-side logger would need to be used per-thread)
• PDB, the Python debugger
  • import pdb; pdb.set_trace() enter PDB programmatically
  • pp obj.__dict__ pretty-print an object as a dict
• Testing
  • Overview (Hitchhiker’s Guide to Python)
  • Testing tools taxonomy
  • unittest, the standard library’s testing framework
    • Test classes subclass unittest.TestCase
    • Individual tests (methods on the subclass) begin test
    • setUp, tearDown, setUpClass, tearDownClass to handle before/after every and before/after all
    • debug instead of run to allow debugger usage and original exception propagation
    • Support for async tests (requires >= Python 3.8)
    • Automatic test discovery
    • Optional test skipping
  • unittest.mock, the standard library’s mocking library
    • Mocks
      • Mock records calls and can be stubbed
      • MagicMock also autogenerates magic methods like __str__
      • AsyncMock returns awaitables
    • Patching
      • Monkey patch objects (functions, classes, contexts), e.g. to replace with mocks

Snippets

• Point = collections.namedtuple('Point', ['x', 'y']); p = Point(11, y=22) create a named tuple
• assert a == b, 'Something went wrong' prescribe an assertion-failure message
• except (IndexError, ValueError) as err catch multiple possible errors
• format(0xdeadbeef, '032b') format to binary number with leading zeros
• from pprint import pprint import default pretty printer
• print(sys.argv[1:]) command-line arguments
• python3 -m http.server start simple HTTP server

Code

• Run asynchronous main function (requires Python 3.7+)

import asyncio

async def main():
    """Asynchronous main function."""
    print('hello')
    await asyncio.sleep(1)
    print('world')

if __name__ == '__main__':
    asyncio.run(main())

• Decorator to indicate and check parent method is being overridden

def overrides(interface_class):
    """
    Allow runtime checking of @override() decorator.

    Usage:
    > @overrides(logging.Handler)
    > def emit(self, record):
    >     pass

    See https://stackoverflow.com/questions/1167617/#answer-8313042.
    """

    override_err = 'Overriden method does not exist in parent.'

    def overrider(method):
        assert method.__name__ in dir(interface_class), override_err
        return method

    return overrider

Virtual machines

Intro

• About (wikipedia)
• Implementation (MIT 6.828)

Vagrant

• Use case: define and interact with development environments
• Virtualization options include Docker, VirtualBox, VMware, and others
• Plugins
  • vagrant-scp SCP between host and VM
    • vagrant scp magma:/var/opt/magma/certs/gateway.key ~/Desktop/magma_protos/gateway.key ; vagrant scp magma:/var/opt/magma/certs/gateway.crt ~/Desktop/magma_protos/gateway.crt
• Snippets
  • vagrant ssh magma -c 'cd magma/lte/gateway && make test' run command over SSH
  • vagrant box list list current images
  • vagrant box add magma_dev_tmp magma_dev_virtualbox.box add image from Packer output
  • vagrant cloud auth login login using info from $ATLAS_TOKEN
  • vagrant cloud auth whoami who I’m logged in as
  • vagrant cloud publish fbcmagma/magma_dev 1.0.1586316960 virtualbox magma_dev_virtualbox.box publish new version

Packer

• Use case: define and build automated machine images
• Notes
  • Waiting for SSH to become available... can take 5-10 minutes
• Snippets
  • packer build magma-dev-virtualbox.json build image from config

Databases

General

• Database engines ranking, with additional info
• Interoperability and standardization: don’t assume full interoperability, though compatibility is possible for a subset of expressions across well-chosen SQL engines
• Transaction isolation levels in common databases (Hermitage)
• Best practices
• Case sensitivity for identifiers
  • Postgres converts unquoted to lowercase
  • MySQL does not change unquoted
• Tools
  • gh-ost: online schema migrations for MySQL

Graph databases

• Gremlin/TinkerPop tutorial

SQLite

• Cheat sheet
• How to corrupt an SQLite database file
• Tools
  • litecli: autocomplete and syntax highlighting for SQLite CLI
• Snippets
  • .help show help
  • .tables list tables
  • .schema ZRECIPE show a table’s schema
  • .backup backup.sqlite create a consistent backup of the DB

Redis

• Cheat sheet
• Snippets
  • redis-cli -p 6380 connect to Redis locally on port 6380
  • keys * find all keys
  • get mobilityd:systemd_status get value of key mobilityd:systemd_status

MySQL

• Cheat sheet
• MariaDB: drop-in-replacement fork of MySQL (comparison)
• Command-line snippets
  • mysql -u magma_dev -p access as user magma_dev, with password prompt
• Snippets
  • show databases;
  • use magma_dev; use magma_dev database
• SQL snippets
  • show tables;
  • describe some_table; show schema for table
  • table some_table; print full table
  • select * from some_table limit 10;
  • select count(*) from some_table; number of rows in some_table
  • select count(distinct key) from certificate_info_db; number of distict values in column key
  • select id,convert(data using utf8) from reservations; convert a blob field data into readable UTF-8
  • select id,convert(data using utf8) from reservations where data like '%testHardwareAssetIds%' limit 1; search e.g. a JSON field for particular key
  • delete from certificate_info; delete all rows
  • delete from states where type='directory_record' and key='id=foo'; delete based on column contents
  • drop table some_table;
  • begin transaction isolation level serializable; begin serializable transaction
  • commit; commit transaction
  • end; rollback transaction

PostgreSQL

• Cheat sheet
• PostgreSQL vs. MySQL
• Job queue with SKIP LOCKED (more)
  • Won’t work / doesn’t make sense with SERIALIZABLE isolation level
• Dump and import
  • Dump
    • Install postgresql-client version to match server
    • pg_dump -U magma -h orc8r.remote.host magma > /tmp/pgdump-prod-$(date +%s).sql dump remote magma database contents
  • Import
    • createdb -U magma_dev -T template0 pgdump-prod-1598488376 create new virgin database
    • psql -U magma_dev pgdump-prod-1598488376 < /var/lib/postgresql/data/pgdump-prod-1598488376.sql import dumped database into created database
    • Note that may also have to create appropriate users/roles for imported database
  • Drop
    • dropdb -U magma_dev pgdump-prod-1598488376 drop database
• CLI snippets
  • psql -U magma_dev access as user magma_dev, with password prompt
• Snippets
  • \l list all databases
  • \c magma_dev use magma_dev database
  • \dt list all tables in current database (\dt+ for more info)
  • \d+ hwid_to_hostname detailed information on table hwid_to_hostname
  • explain select task_id from test_task_bag limit 1 for update skip locked; explain query plan
  • select key, encode(value, 'escape') as value from directoryd_blobstore where type='hwid_to_hostname'; decode bytes to string using encode string function
  • select * from cfg_entities \g out.table output query to file
  • \copy (select * from cfg_entities) to out.csv csv header output query to CSV file

C++

• Google C++ style guide
• constexpr indicate a variable or function value can be evaluated at compilation time

C

Intro

• Resources
• Style: (format) (portability) (header files)

Language

• C operator precedence: highest precedence are ++ -- () [] . ->
• static keyword inside a function: visible in function’s scope, but extends lifetime across all calls to function (and stored in the usual data or bss sections)
• main function signature: int main (int argc, char *argv[], char *envp[])
  • argc number of command-line args
  • argv null-terminated array of C strings representing the passed command-line args
    • First arg: running program name
  • envp null-terminated array of environment variable strings
    • Each take the format of "variable=value"
    • Less portable to include envp
    • Stale to the time of process fork
    • Not necessary, less usable than getenv
• Error handling
  • perror(char *msg): print msg and error message corresponding to errno
  • exit(EXIT_FAILURE) portable exit with failure status
    • Call functions registered as on-exit, flush and close streams, remove tmp files
• Print individual bits
• Memory barrier, memory fence: enforce ordering constraint on memory operations (valuable for concurrent programs)
• Labels are function-local by default
• Define a small function in header file with static inline modifier
• Declare a struct with typedef: typedef struct { int x, y; } Point;
• Size of an array, also works for variable-sized arrays: int arr[arr_size]; sizeof(arr);
• Zeroize a struct (initialization also works for static-sized arrays)
  • On initialization: struct vertex_t vertex = {0};
  • On-demand: memset(&vertex, 0, sizeof(vertex_t));
• Type punning preferred with unions (or memcpy)
• isatty: test whether file descriptor refers to a terminal
• Avoid zombie processes: wait(), double fork(), or ignore in parent with signal(SIGCHLD, SIG_IGN)
• How to write shared libraries
• Libevent: event notification library for event-driven programming
• Protothreads: lightweight, stackless “threads” for use especially in embedded systems

Standards

• C standard
  • Aka ANSI C, ISO C
  • Standardized definition of the language, as jointly published by the ISO and IEC
  • Recent versions include (C18) (C11) (C99)
  • Where to find specific versions
    • Full text of C99 standard (html) (pdf)
• Standard library
  • C standard library is the standard library specified in ISO C (headers)
    • Aka ISO C library, libc
    • glibc is the prominent implementation of libc (source)
  • POSIX: Portable Operating System Interface (headers) (wikipedia)
  • POSIX is a superset of ISO C that also defines methods for interacting with the OS

    

Compilation

• Compilation process: (source code) (assembly) (object code) (executable)

  

• ELF: Executable and Linkable Format: standardized, cross-platform file format for executables, object code, shared libraries, and core dumps

  

• GCC: GNU Compiler Collection: compiler system handling languages like C and C++
• How the #include directive works: quotes first search current working directory, otherwise "" and <> are largely similar
• echo | gcc -xc -E -v - print the include paths used by gcc/g++
• gcc --print-file-name=libc.a look for a library, e.g. libc.so or libc.a
• Cross compiling

Preprocessor

• Reference (GCC’s C preprocessor)

• Pragmas: provide additional information to the particular compiler, beyond language-standardized directives

  • #pragma once modern alternative for providing #include guards (stackoverflow)

• Current function name: __func__

• Variadic macros: macros with a variable number of arguments

  • Use a trick to send zero arguments to a variadic macro

• Trigraphs: nine triplets, each beginning with ??, which can optionally be replaced by their alternative single-character representations; e.g. ??!??! represents ||, the logical or operator

• Stringification with #

   // Will actually print EXP
   #define WARN_IF(EXP) \
   do { if (EXP) \
           fprintf (stderr, "Warning: " #EXP "\n"); } \
   while (0)
  
   WARN_IF(x == 0);  // prints "Warning: x == 0"

• Token concatenation with ##

  #define macro_start i ## n ## t m ## ain(void)
  macro_start;  // -> int main(void);

• Token expansion: in function-like macros, if either of the tokens next to a ## is a parameter name, it is replaced by its actual argument before ## executes

• Debug print:

  #ifdef DEBUG
  #define DEBUG_PRINT(...) fprintf( stderr, __VA_ARGS__ )
  #else
  #define DEBUG_PRINT(...) do { } while (0)
  #endif  // DEBUG

Make

• Intro
• Reference
  • Manual
  • Managing projects
  • Tips and tricks
• Usage
  • Automatic variables
    • $* the stem which an implicit rule matches
  • Double-colon rules run a single matching rule instead of all matching rules for a target
  • .PHONY targets if a file with the target’s name exists, prefer the target over the file
    • Can append via successive .PHONY designations
• Snippets
  • make -j make non-dependent executables/libraries in parallel
  • make --debug=v print debug info
  • $(PROTO_LIST): %_protos: build_list static pattern rule
  • Self-documenting Makefile make help: ## Print help text @grep -Eh '^[a-zA-Z_-]+:.*?## .*$$' $(MAKEFILE_LIST) | sort | awk 'BEGIN {FS = ":.*?## "}; {printf "\033[36m%-30s\033[0m %s\n", $$1, $$2}'
  • Print all variables make printvars: @$(foreach V,$(sort $(.VARIABLES)), \ $(if $(filter-out environment% default automatic, \ $(origin $V)),$(warning $V=$($V) ($(value $V)))))

CMake

• Tutorial
• Set up a debug build
  • From CMakeLists.txt: set(CMAKE_BUILD_TYPE Debug)
  • From command line: cmake -DCMAKE_BUILD_TYPE=Debug <...>
• Hook up CMake with Vim’s YouCompleteMe via a compilation database
  1. set(CMAKE_EXPORT_COMPILE_COMMANDS ON)
  2. Allow YCM to find the compile_commands.json file by either moving the file into some directory above the working directory (as in the link above), or utilizing a .ycm_extra_conf.py to point to the file
• Remove unused header includes with include-what-you-use, which can be integrated with CMake

Sockets

• Overview (Beej’s Guide to Network Programming)
• glibc manual

  

• Debugging
  • If recv is mysteriously reading zero bytes, despite peer not closing connection, make sure the provided buffer length is non-zero
  • Frozen program may be holding a socket
    • Check for whether local port is bound: nmap -p - localhost
    • Force-kill the parent process: kill -9 <pid>

GDB

Overview

• The GNU Project debugger, allows start, stop, lookaround, and modification
• Supports languages like C, C++, Go, Rust
• rr: related debugger with deterministic replay and reverse execution

Intro

• Cheat sheet
• Intro (Beej’s Quick Guide to GDB)
• Tutorial
• Documentation
• Intel’s x86 architecture manual
• Better GUI options
  • gdb-dashboard: in-place .gdbinit upgrade (doesn’t play well with gdb’s TUI)
  • gdbgui: browser-based, full-featured

Commands

Startup

• Pass command-line args

  • From command line: gdb --args ./executable arg1 arg2 arg3
  • From gdb: (gdb) r arg1 arg2 arg3

• Automatically run executable: gdb -ex run --args ./executable arg1 arg2 arg3

• Core dumps

  • Message at program termination: Segmentation fault (core dumped)
  • Load core dump file: gdb path/to/binary path/to/core
  • Fix missing core dump, can be caused by issue on Ubuntu: ulimit -c unlimited

• .gdbinit init script: run commands on gdb startup

  • Can be placed in home directory or in current working directory
  • Fix for occasional issue with loading .gdbinit from local dir

• Interrupt execution programmatically

  #include <signal.h>
  raise(SIGINT);

• Save command history across sessions: echo 'set history save on' >> ~/.gdbinit && chmod 600 ~/.gdbinit

Moving around

• <C-c> interrupt execution
• c continue execution
• si step a machine instruction (s for source instruction)
• ni same as si but step over function calls (n for source instructions)
• finish run until current function finishes
• return discard current stack frame and return immediately
• break 120 if x == 3
  • 120 can be <line>, <file>:<line>, <function>
  • if x == 3 can be any conditional, or omitted
  • info break info on all breakpoints
  • disable 3 disable breakpoint 3
  • enable 3 enable breakpoint 3
  • del 3 delete breakpoint 3
• advance 120 run until reach line 120 (or some location)

Looking around (tools)

• General information
  • wh open TUI
  • bt stack backtrace
  • list surrounding source lines
  • frame current line, function, and args; change stack frame
  • up/down move up/down stack frame
  • info
    • Examining the symbol table
    • i r current registers
    • i address <symbol> address of <symbol>
    • i variables variables known in current scope
    • i locals local variables
    • i args args to current stack frame
    • i thread running threads
• x examine memory
  • x print raw contents of memory
    • x/nfu addr
      • n repeat count
      • f display format: hex, decimal, unsigned decimal, octal, two (binary), address, char, float, string, zero-padded hex, raw
      • u unit size: byte, halfword (2 bytes), word (4 bytes), giant word (8 bytes)
    • x/s <addr> print full null-terminated string starting at <addr>
    • x/5i $eip print next 5 instructions w/ next instruction at top
• p(rint) print evaluated c expression
  • p/x $cs print content of $cs register
• Using the TUI
  • wh open TUI
  • layout <name> change layout, where <name> can take a superset of:
    • src source, command
    • split source, assembly, command
    • asm assembly, command
    • regs add registers to current layout
  • focus <name> set main focus to <name> sub-window, similar to layout above
  • wh <name> <count> change window height of <name> by <count>, where count can be positive or negative
  • tui disable or <C-x> a disable the TUI

Looking around (tips)

• Examine stack
  • info frame print human-readable information about the active stack
  • p ($ebp-$esp)/4 + 2 print the “length” of the stack necessary to see the stack down to the saved ebp and ret addr
  • x/16xw $esp print the full stack, where 16 is the output from the above command
  • x/2xw $ebp print saved ebp and saved ret addr
• Examine text segment
  • info line info about the current line of the active frame
  • disas print dissembled x86 instructions starting at eip
  • disas /r $eip,+24 print 24 instructions, including each instructions specific bytes
  • disas /s $eip,+24 print 24 instructions, including relevant source code
• Printing information
  • display/<format> <expr> to display <expr> after every command to gdb

Breakpointing

• break <linenum> break just before <linenum> executes in the current file
• break <filename>:<linenum> break just before <linenum> executes in <filename>
• until <linenum> step until source’s <linenum> is reached, or would exit current stack
• watch <expr> to break when the evaluated value of <expr> changes
  • set can-use-hw-watchpoints 0 may have to enable software-only watchpoints (to function properly on a VMWare VM?)
• delete display clear all auto-displays

Editing memory

• Can use either print or set to set values/edit memory during debugging
• set reqpath = "foo" where reqpath is a char*, sets the reqpath string to "foo"
• set {int}0xbffff59c = 0x10 set the memory at address 0bxffff59c to the value 0x10

Miscellanea

• If some symbols seem to be missing, check the compiler’s optimization level
• set $esp += 4 increment $esp register by 4
• define <command> define a user alias to <command>
• target record-full record all actions so can reverse debug
• Debugging threads
• Debugging forks: set follow-fork-mode child
• Specify a location: br * <addr> break at addr

Security

Overview

• Overview of cryptography
• Awesome cryptography
• Textbooks
  • Introduction to Cryptography
  • Handbook of Applied Cryptography
• Cryptographic key
  • Key types
• TinySec, an example use-case with great explanations of some cryptographic topics

Information security

• Information security overview
• Exact origins of “CIA triad” terminology unknown

Security tools

• Practical online security
  • Password strength checker (entropy) (time to crack)
  • Strong password generator (Diceware)
  • 2FA: set up two-factor authentication for important websites
  • Firefox Monitor: see if you’ve been part of an online data breach
• Additional tools
  • Browser fingerprinting (AmIUnique)
  • Tor (wikipedia)
    • How HTTPS and Tor work together
  • Keybase
  • Qubes OS
  • Keychain Access (MacOS)
    • Identity preferences map mTLS client certificates to domains

ASN.1

• Abstract Syntax Notation One is an interface description language (objects-only) allowing data structure definitions in support of platform-agnostic serialization, used extensively in cryptographic applications
  • ASN.1 specifies abstract objects
  • Encodings translate the abstract objects into a stream of bytes; DER, a subset of BER, gives a unique encoding value to each ASN.1 object, a useful quality for cryptographic purposes (all encodings)
  • Encoding of particular object generally follows the type-length-value (TLV) scheme
  • Primitive types list
• Complete information 🌟
• Resources (about) (intro) (specification intro) (cheat sheet)
• Playground
  • Denote octet string values as 'deadbeef'H
• Object ID (OID) repository
• pyasn1: an ASN.1 parser written in Python (github) (tutorial)

OpenSSL

• Home (wikipedia)
• Common commands (more 1) (more 2)
• OpenSSL Cookbook
• Common file types
  • .pem Privacy Enhanced Mail, a Base64-encoded bucket, where file contents are indicated in the header/footer lines
  • .key any type of key, usually a private key
  • .csr certificate signing request
  • .pfx PKCS #12, a generic archive format; often: encrypted and signed bucket containing a private key and associated X.509 certificate/chain
  • Common certificate filename extensions: .pem, .cer, .crt, .der, .pfx
• Common tasks
  • Set up a CA (more)
  • Implement mutual TLS authentication (mTLS)
• Snippets
  • openssl pkcs12 -in admin_operator.pfx -nocerts -nodes -out out.key extract private key from PKCS #12 (or -nokeys to extract just certificates) (man) (dealing with bag attributes)
  • openssl rsa -text -noout -in out.key examine RSA key (man) (more)
  • openssl x509 -text -noout -in certifier.pem examine X.509 certificate (man)
  • openssl req -noout -text -in controller.csr examine PKCS #10 certificate signing request (man)
  • openssl verify -CAfile ca.pem -untrusted intermediate.cert.pem cert.pem verify certificate chain
  • openssl asn1parse -i -in out.crt examine ASN.1-parsed fields
  • openssl s_client -CAfile /var/opt/magma/certs/rootCA.pem -connect bootstrapper-controller.magma.test:9444 debug SSL handshake (man)

Let’s Encrypt

• Overview

  

• Getting started
• Certbot automatically generate TLS certificates

Security primitives

• Message authentication code (MAC): symmetric analog to the (asymmetric) digital signature; a keyed hash that uses shared symmetric key to tag/validate the conveyed message to provide message integrity and authentication
• Block cipher modes (about) (specs) (discussion)
  • Confidentiality modes: ECB, CBC, CFB, OFB
  • Authentication mode: CMAC
  • Authenticated encryption mode: CCM
  • High-throughput authenticated encryption mode: GCM
  • Methods for key wrapping: KW, KWP, TDEA
  • And a few others
• Hash chain: successive application of one-way hash function to a piece of data
  • In password storage/authentication: server computes and stores h^1000(password), and user successively authenticates by sending h^999(password), h^998(password), …

Keying

• Key exchange: sharing secrets between two or more parties
  • Group key: key shared between group of users, typically for individual decrypting of a broadcast message
  • Pre-shared key (PSK): secret shared between parties before first use
    • TLS can use pre-shared keys to establish a TLS connection
    • PSKs are associated with an identity (a UTF-8-encoded string), so e.g. a client would store its own (ID, PSK) pair while a server would store multiple (ID, PSK) pairs
• Key derivation function (KDF): derive one or more secret keys from some input value, e.g. to generate session keys from a master key or a symmetric key from a password

Public-key cryptography

• PKI overview
• Discrete logarithm problem: deciding S given (g, p, g^S mod p) (generator, prime, key fragment) being a hard problem
• Diffie-Hellman key exchange (DH)
  • Diffie-Hellman considered a “public-key technology” since it has public and private components; the algorithm itself doesn’t encrypt anything
  • Using ephemeral keys with Diffie-Hellman allows for perfect forward secrecy because the “private” DH keys are stored only for the duration of the session, then discarded (in fixed DH, the public parameter is embedded in the node’s certificate)
• Perfect forward secrecy
  • Consider all past messages recorded, available, and algorithms fully specified: could the sender/receiver collaborate to recover the plaintext of these past messages? If yes, then the strategy does not allow for PFS
  • Assume that anything sent using long-term public keys (RSA public key, DH public parameter, etc.) is cleartext (description)
  • Focus is the ephemerality of the DH keypairs (which are used to generate the session key), rather than ephemerality the session keys (which are always ephemeral)
  • Example of non-forward secret strategy: generate a session key, encrypt it with receiver’s public key, and send to receiver (key transport)
  • DH vs. RSA for PFS
  • Key-agreement protocols where both parties influence the final derived key are the only way to implement PFS
• Key encapsulation: method for choosing an asymmetrically-transported symmetric key which doesn’t require the use of a padding scheme
  • Optimal Asymmetric Encryption Padding (OAEP), a padding scheme often used with RSA encryption
• Public key fingerprint: encode key into bytes, hash it, then pretty print it as e.g. a hexadecimal string; produces a shorter, human-readable version of the key (and optional additional data like name of key holder)
• Public key certificate: node’s public key, identifying information, and digital signature of a relevant certificate authority (file formats)

• Digital signature: first hash the message (e.g. a certificate’s identifying information) into a digest, pad, then sign with the private key (e.g. by raising to the private key, mod an appropriate N)
  • A digest is the output of a hash function, descriptively relevant when used to verify a fingerprint without recovering e.g. the fingerprint’s source key
  • Key signature is one party signing another party’s key fingerprint
  • Signatures can also be blind, using a blinding function first before signing
• Public key infrastructure binds public keys to identities using certificates issued by a certificate authority
  • CA (certificate authority): store, issue, sign certificates
  • RA (registration authority): verify identities of entities requesting certificates
  • VA (validation authority): verify that a certificate is valid, e.g. by distributing certificate revocation lists (CRLs) or implementing the Online Certificate Status Protocol (OCSP)

• Hybrid cryptosystem combines functionality of public-key with efficiency of symmetric-key
  • Key encapsulation scheme uses public-key
  • Data encapsulation uses symmetric-key
  • Majority of cryptosystems use this method–e.g. TLS, OpenPGP, etc.
• Pretty Good Privacy (PGP)
  • Hybrid cryptosystem where symmetric “session” key is the only data encrypted with receiver’s public key
  • Follows OpenPGP open specification

• Post-quantum cryptography
  • Supersingular isogeny key exchange: post-quantum analog to Diffie-Hellman